1 / 50

04_Logistic_Regression

linear regression

HATHIRAM
Download Presentation

04_Logistic_Regression

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Logistic Regression CS60010: Deep Learning Abir Das IIT Kharagpur Jan 22, 23 and 24, 2020

  2. Logistics Agenda Linear Regression Logistic Regression Some Logistics Related Information § This Friday (Jan 24), no paper will be presented. It will be a regular lecture. § The first surprise quiz is today!! Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 2 / 35

  3. Logistics Agenda Linear Regression Logistic Regression Surprise Quiz 1 § The duration of the test is 10 minutes. § Question 1: Find the eigenvalues of the following matrix A. Clearly mention if you are making any assumption. [2 Marks]  −1 § Question 2: Consider the half-space given by the set of points S = {x ∈ Rd|aTx ≤ b}. Prove that the halfspace is convex. [3 Marks]  2 1 0 3 0 0 0 1   Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 3 / 35

  4. Logistics Agenda Linear Regression Logistic Regression Surprise Quiz 1: Answer Keys § Question 1: Find the eigenvalues of the following matrix A. Clearly mention if you are making any assumption.  −1 Use the property of eigenvalues of a triangular matrix. § Question 2: Consider the half-space given by the set of points S = {x ∈ Rd|aTx ≤ b}. Prove that the halfspace is convex. : If x,y belong to S, then aTx ≤ b and aTy ≤ b. Now, for 0 ≤ θ ≤ 1, aT{θx + (1 − θ)y} = θaTx + (1 − θ)aTy ≤ θb + (1 − θ)b = b  2 1 0 3 0 0 0 1   Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 4 / 35

  5. Logistics Agenda Linear Regression Logistic Regression Agenda § Understand regression and classification with linear models. § Brush-up concepts of maximum likelihood and its use to understand linear regression. § Using logistic function for binary classification and estimating logistic regression parameters. Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 5 / 35

  6. Logistics Agenda Linear Regression Logistic Regression Resources § The Elements of Statistical Learning by T Hastie, R Tibshirani, J Friedman. [Link] [Chapter 3 and 4] § Artificial Intelligence: A Modern Approach by S Russell and P Norvig. [Link] [Chapter 18] Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 6 / 35

  7. § We have a set of N observations of y as {y1,y2,··· ,yN} and the corresponding input variables {x1,x2,··· ,xN}. %(#) !(#) Logistics Agenda Linear Regression Logistic Regression Linear Regression § In a regression problem we want to find the relation between some input variables x and output variables y, where x ∈ Rdand y ∈ R. § Inputs are also often referred to as covariates, predictors and features; while outputs are known as variates, targets and labels. § Examples of such input-output pairs can be I {Outside temperature, People inside classroom, target room temperature | Energy requirement} I {Size, Number of Bedrooms, Number of Floors, Age of the Home | Price} Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 7 / 35

  8. Logistics Agenda Linear Regression Logistic Regression Linear Regression § In a regression problem we want to find the relation between some input variables x and output variables y, where x ∈ Rdand y ∈ R. § Inputs are also often referred to as covariates, predictors and features; while outputs are known as variates, targets and labels. § Examples of such input-output pairs can be I {Outside temperature, People inside classroom, target room temperature | Energy requirement} I {Size, Number of Bedrooms, Number of Floors, Age of the Home | Price} § We have a set of N observations of y as {y1,y2,··· ,yN} and the corresponding input variables {x1,x2,··· ,xN}. %(#) !(#) Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 7 / 35

  9. Logistics Agenda Linear Regression Logistic Regression Linear Regression § The input and output variables are assumed to be related via a relation, known as hypothesis. b y = hθ(x), where θ is the parameter § The goal is to predict the output variableb y∗= f(x∗) for an arbitrary § Let us start with scalar inputs (x) and scalar outputs (y). vector. value of the input variable x∗. Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 8 / 35

  10. § Gradient of the cost function w.r.t. θ0: N X ?hθ(x(i)) − y(i)? J(θ0,θ1) θ0 1 N = i=1 § Gradient of the cost function w.r.t. θ1: J(θ0,θ1) N X ?hθ(x(i)) − y(i)?x(i) 1 N = θ1 i=1 § Apply your favorite gradient based optimization algorithm. Logistics Agenda Linear Regression Logistic Regression Univariate Linear Regression § hypothesis: hθ(x) = θ0+ θ1x. § Cost Function: Sum of squared errors. %(#) N X ?hθ(x(i)) − y(i)?2 1 J(θ0,θ1) = 2N i=1 !(#) § Optimization objective: find model parameters (θ0,θ1) that will minimize the sum of squared errors. Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 9 / 35

  11. Logistics Agenda Linear Regression Logistic Regression Univariate Linear Regression § hypothesis: hθ(x) = θ0+ θ1x. § Cost Function: Sum of squared errors. %(#) N X ?hθ(x(i)) − y(i)?2 1 J(θ0,θ1) = 2N i=1 !(#) § Optimization objective: find model parameters (θ0,θ1) that will minimize the sum of squared errors. § Gradient of the cost function w.r.t. θ0: J(θ0,θ1) θ0 N § Gradient of the cost function w.r.t. θ1: J(θ0,θ1) θ1 N N X ?hθ(x(i)) − y(i)? 1 = i=1 N X ?hθ(x(i)) − y(i)?x(i) 1 = i=1 § Apply your favorite gradient based optimization algorithm. Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 9 / 35

  12. Logistics Agenda Linear Regression Logistic Regression Univariate Linear Regression § These being linear equations of θ, have a unique closed form solution too. N P N i=1 y(i)x(i)−? N P N P 1 N i=1 x(i)?? N P x(i)?2 N X y(i)? N i=1 i=1 i=1 θ1= ?x(i)?2−? N P N X i=1 ? x(i)? y(i)− θ1 θ0= i=1 Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 10 / 35

  13. Logistics Agenda Linear Regression Logistic Regression Multivariate Linear Regression § We can easily extend to multivariate linear regression problems, where x ∈ Rd § hypothesis: hθ(x) = θ0+ θ1x1+ θ2x2+ ··· + θdxd. For convenience of notation, define x0= 1. § Thus h is simply the dot product of the parameters and the input vector. hθ(x) = θTx § Cost Function: Sum of squared errors. N X ?θTx(i)− y(i)?2 1 J(θ) = J(θ0,θ1,··· ,θd) = (1) 2N i=1 § We will use the following to write the cost function in a compact matrix vector notation hθ(x) = θTx = xTθ Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 11 / 35

  14. Logistics Agenda Linear Regression Logistic Regression Multivariate Linear Regression          = b y = Xθ    = θ0 θ1 θ2 ... θd x(1) 0 x(2) 0 ... x(N) 0 x(1) 1 x(2) 1 ... x(N) 1 x(1) 2 x(2) 2 ... x(N) 2 x(1) d x(2) d b y(1) ... b y(N) hθ(x(1)) hθ(x(2)) ... hθ(x(N)) ··· ··· ... ···         b y(2) (2)   x(N) d Here, X is a N × (d + 1) matrix with each row an input vector. b y is a N length vector of the outputs in the training set. Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 12 / 35

  15. Logistics Agenda Linear Regression Logistic Regression Multivariate Linear Regression § Eqn. (1), gives, N N X X ?θTx(i)− y(i)?2= 2N||b y − y||2 2N 1 2N 1 2N ?b y(i)− y(i)?2 1 1 (3) J(θ) = 2N 2N i=1 i=1 ?b y − y?T?b y − y? 2N 1 1 = 2= 2N ?Xθ − y?T?Xθ − y?= ?θT?XTX?θ −?XTy?Tθ −?XTy?Tθ + yTy? ?θT?XTX?θ − 2?XTy?Tθ + yTy? ?θT?XTX?θ − θTXTy − yTXθ + yTy? 1 1 = = = Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 13 / 35

  16. § This gives a closed form solution, but another option is to use iterative solution (just like the univariate case). N X ?hθ(x(i)) − y(i)?x(i) ∂J(θ) ∂θj 1 N = j i=1 Logistics Agenda Linear Regression Logistic Regression Multivariate Linear Regression § Equating the gradient of the cost function to 0, ?2XTXθ − 2XTy + 0?= 0 XTXθ − XTy = 0 θ =?XTX?−1XTy 1 ∇θJ(θ) = 2N (4) Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 14 / 35

  17. Logistics Agenda Linear Regression Logistic Regression Multivariate Linear Regression § Equating the gradient of the cost function to 0, ?2XTXθ − 2XTy + 0?= 0 XTXθ − XTy = 0 θ =?XTX?−1XTy 1 ∇θJ(θ) = 2N (4) § This gives a closed form solution, but another option is to use iterative solution (just like the univariate case). N X ?hθ(x(i)) − y(i)?x(i) ∂J(θ) ∂θj 1 N = j i=1 Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 14 / 35

  18. Logistics Agenda Linear Regression Logistic Regression Multivariate Linear Regression § Iterative Gradient Descent needs to perform many iterations and need to choose a stepsize parameter judiciously. But it works equally well even if the number of features (d) is large. § For the least square solution, there is no need to choose the step size parameter or no need to iterate. But, evaluating?XTX?−1can be slow if d is large. Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 15 / 35

  19. § An alternate view considers the following. I y(i)are generated from the x(i)following a underlying hyperplane. I But we don’t get to “see” the generated data. Instead we “see” a noisy version of the y(i)’s. I Maximum likelihood (or in general, probabilistic estimation) models this uncertainty in determining the data generating function. Logistics Agenda Linear Regression Logistic Regression Linear Regression as Maximum Likelihood Estimation § So far we tried to fit a “straightline” (“hyperplane” to be more precise) for linear regression problem. § This is, in a sense, a “constrained” way of looking at the problem. Datapoints may not be perfectly fit to the hyperplane, but “how uncertain” they are from the hyperplane is never considered. Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 16 / 35

  20. Logistics Agenda Linear Regression Logistic Regression Linear Regression as Maximum Likelihood Estimation § So far we tried to fit a “straightline” (“hyperplane” to be more precise) for linear regression problem. § This is, in a sense, a “constrained” way of looking at the problem. Datapoints may not be perfectly fit to the hyperplane, but “how uncertain” they are from the hyperplane is never considered. § An alternate view considers the following. I y(i)are generated from the x(i)following a underlying hyperplane. I But we don’t get to “see” the generated data. Instead we “see” a noisy version of the y(i)’s. I Maximum likelihood (or in general, probabilistic estimation) models this uncertainty in determining the data generating function. Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 16 / 35

  21. Logistics Agenda Linear Regression Logistic Regression Linear Regression as Maximum Likelihood Estimation § Thus data are assumed to be generated as follows. y(i)= hθ(x(i)) + ?(i) where ?(i)is an additive noise following some probability distribution. § So,?x(i),y(i)?’s form a joint distribution. probability distribution is parameterised by some additional parameters (e.g., Gaussian with 0 mean and covariance σ2). § Then find the parameters (both θ and σ2) that is “most likely” to generate the data. § The idea is to assume a probability distribution on the noise and the Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 17 / 35

  22. Logistics Agenda Linear Regression Logistic Regression Recall: Cost Function Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 18 / 35

  23. Logistics Agenda Linear Regression Logistic Regression Alternate View: “Maximum Likelihood” Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 19 / 35

  24. Logistics Agenda Linear Regression Logistic Regression Maximum Likelihood: Example § Intuitive example: Estimate a coin toss I have seen 3 flips of heads, 2 flips of tails, what is the chance of head (or tail) of my next flip? § Model: Each flip is a Bernoulli random variable x. x can take only two values: 1(head), 0(tail) ? θ, if x = 1 if x = 0 p(x|θ) = (5) 1 − θ, where, θ ∈ [0,1], is a parameter to be defined from data § We can write this probability more succinctly as p(x|θ) = θx(1 − θ)1−x (6) Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 20 / 35

  25. § Intuitively, one could assume that it is the number of heads we got divided by the total number of coin throws. § We will prove in the following that the intuition in this case is correct, by proving that the guess θ =P § Then the joint probability is Y ixi/n is the “most likely” value for the real θ. P n−P xi(1 − θ) xi (7) f(x1,....,xn;θ) = f(xi;θ) = θ i i i Logistics Agenda Linear Regression Logistic Regression Maximum Likelihood: Example § Let us now assume, that we have flipped the coin a few times and got the results x1,...,xn, which are either 0 or 1. The question is what is the value of the probability θ? Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 21 / 35

  26. § We will prove in the following that the intuition in this case is correct, by proving that the guess θ =P § Then the joint probability is Y ixi/n is the “most likely” value for the real θ. P n−P xi(1 − θ) xi (7) f(x1,....,xn;θ) = f(xi;θ) = θ i i i Logistics Agenda Linear Regression Logistic Regression Maximum Likelihood: Example § Let us now assume, that we have flipped the coin a few times and got the results x1,...,xn, which are either 0 or 1. The question is what is the value of the probability θ? § Intuitively, one could assume that it is the number of heads we got divided by the total number of coin throws. Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 21 / 35

  27. § Then the joint probability is P n−P Y xi(1 − θ) xi (7) f(x1,....,xn;θ) = f(xi;θ) = θ i i i Logistics Agenda Linear Regression Logistic Regression Maximum Likelihood: Example § Let us now assume, that we have flipped the coin a few times and got the results x1,...,xn, which are either 0 or 1. The question is what is the value of the probability θ? § Intuitively, one could assume that it is the number of heads we got divided by the total number of coin throws. § We will prove in the following that the intuition in this case is correct, by proving that the guess θ =P ixi/n is the “most likely” value for the real θ. Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 21 / 35

  28. Logistics Agenda Linear Regression Logistic Regression Maximum Likelihood: Example § Let us now assume, that we have flipped the coin a few times and got the results x1,...,xn, which are either 0 or 1. The question is what is the value of the probability θ? § Intuitively, one could assume that it is the number of heads we got divided by the total number of coin throws. § We will prove in the following that the intuition in this case is correct, by proving that the guess θ =P § Then the joint probability is Y ixi/n is the “most likely” value for the real θ. P n−P xi(1 − θ) xi (7) f(x1,....,xn;θ) = f(xi;θ) = θ i i i Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 21 / 35

  29. Logistics Agenda Linear Regression Logistic Regression Maximum Likelihood: Example § We now want to find the θ which makes this probability the highest. § It is easier to maximize the log of the joint probabilities log L(θ) =P result, since the log is monotonously increasing. § As we may remember, maximizing a function means setting its first derivative to 0. P θ (1 − θ)P P θ(1 − θ) P n Abir Das (IIT Kharagpur) CS60010 xilog θ + (n −P xi)log (1 − θ), which yields the same i i (n −P 1 − θ xi− θn + θP θ(1 − θ) xi− θn = 0 xi xi) ∂log L(θ) ∂θ i i − = xi i i = i = xi i =⇒ θ = (8) Jan 22, 23 and 24, 2020 22 / 35

  30. Finding the θ that maximizes the likelihood is equivalent to moving the Gaussian until the product of 3 green bars (likelihood)is maximized. Slide Motivation: Nando de Freitas [Link] Logistics Agenda Linear Regression Logistic Regression Maximum Likelihood Estimation We have n = 3 data points y1= 1,y2= 0.5,y3= 1.5, which are independent and Gaussian with unknown mean = θ and variance = 1 : yi∼ N(θ,1) with likelihood P(y1,y2,y3;θ) = P(y1;θ)P(y2;θ)P(y3;θ). Consider two guesses of θ, 1 and 2.5. Which has higher likelihood (probability of generating the three observations)? Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 23 / 35

  31. Logistics Agenda Linear Regression Logistic Regression Maximum Likelihood Estimation We have n = 3 data points y1= 1,y2= 0.5,y3= 1.5, which are independent and Gaussian with unknown mean = θ and variance = 1 : yi∼ N(θ,1) with likelihood P(y1,y2,y3;θ) = P(y1;θ)P(y2;θ)P(y3;θ). Consider two guesses of θ, 1 and 2.5. Which has higher likelihood (probability of generating the three observations)? Finding the θ that maximizes the likelihood is equivalent to moving the Gaussian until the product of 3 green bars (likelihood)is maximized. Slide Motivation: Nando de Freitas [Link] Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 23 / 35

  32. Logistics Agenda Linear Regression Logistic Regression Maximum Likelihood Estimation of model parameters θ § In general, we have observations, D = {u(1),u(2),··· ,u(N)} § We assume data is generated by some distribution U ∼ p(U;θ) § Compute the likelihood function N Y p(u(i);θ) ← Likelihood Function L(θ) = (9) i=1 L(θ) n X θML= argmax θ logp(u(i);θ) ← Log Likelihood (10) = argmax θ i=1 § log(f(x)) is monotonic/ increasing, same argmax as f(x) Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 24 / 35

  33. § Let us compute the likelihood. N Y N Y p(y|X;θ,σ2) = p(y(i)|x(i);θ,σ2) i=1 ? y(i)−θTx(i)?2 ? 1 (2πσ2)−1 2e− = 2σ2 i=1 y(i)−θTx(i)?2 N P ? 1 − = (2πσ2)−N 2σ2 2 e i=1 y−Xθ?T? y−Xθ? 1 = (2πσ2)−N 2 e− (11) 2σ2 Logistics Agenda Linear Regression Logistic Regression Maximum Likelihood for Linear Regression § Let us assume that the noise is Gaussian distributed with mean 0 and variance σ2 y(i)= hθ(x(i)) + ?(i)= θTx(i)+ ?(i) § Noise ?(i)∼ N(0,σ2) and thus y(i)∼ N(θTx(i),σ2). Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 25 / 35

  34. Logistics Agenda Linear Regression Logistic Regression Maximum Likelihood for Linear Regression § Let us assume that the noise is Gaussian distributed with mean 0 and variance σ2 y(i)= hθ(x(i)) + ?(i)= θTx(i)+ ?(i) § Noise ?(i)∼ N(0,σ2) and thus y(i)∼ N(θTx(i),σ2). § Let us compute the likelihood. N Y N Y p(y|X;θ,σ2) = p(y(i)|x(i);θ,σ2) i=1 ? y(i)−θTx(i)?2 ? 1 (2πσ2)−1 2e− = 2σ2 i=1 y(i)−θTx(i)?2 N P ? 1 − = (2πσ2)−N 2σ2 2 e i=1 y−Xθ?T? y−Xθ? 1 = (2πσ2)−N 2 e− (11) 2σ2 Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 25 / 35

  35. § Maximizing the likelihood w.r.t. θ means maximizing −?y − Xθ?T?y − Xθ?which in turn means minimizing § Note the similarity with what we did earlier. § Thus linear regression can be equivalently viewed as minimizing error sum of squares as well as maximum likelihood estimation under zero mean Gaussian noise assumption. ?y − Xθ?T?y − Xθ?. Logistics Agenda Linear Regression Logistic Regression Maximum Likelihood for Linear Regression § So we have got the likelihood as, p(y|X;θ,σ2) = (2πσ2)−N § The log likelihood is l(θ,σ2) = −N ? y−Xθ?T? y−Xθ? 1 2 e− 2σ2 ?y − Xθ?T?y − Xθ? 1 2log(2πσ2) − 2σ2 Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 26 / 35

  36. Logistics Agenda Linear Regression Logistic Regression Maximum Likelihood for Linear Regression § So we have got the likelihood as, p(y|X;θ,σ2) = (2πσ2)−N § The log likelihood is l(θ,σ2) = −N ? y−Xθ?T? y−Xθ? 1 2 e− 2σ2 ?y − Xθ?T?y − Xθ? 1 2log(2πσ2) − 2σ2 § Maximizing the likelihood w.r.t. θ means maximizing −?y − Xθ?T?y − Xθ?which in turn means minimizing § Note the similarity with what we did earlier. § Thus linear regression can be equivalently viewed as minimizing error sum of squares as well as maximum likelihood estimation under zero mean Gaussian noise assumption. ?y − Xθ?T?y − Xθ?. Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 26 / 35

  37. Logistics Agenda Linear Regression Logistic Regression Maximum Likelihood for Linear Regression § In a similar manner, the maximum likelihood estimate of σ2can also be calculated. Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 27 / 35

  38. Logistics Agenda Linear Regression Logistic Regression Classification § y ∈ {0,1}, where 0 : “Negative class” (e.g., benign tumor), 1 : “Positive class” (e.g., malignant tumor) § Some more examples: I Email: Spam/ Not Spam? I Video: Viral/Not Viral? I Tremor: Earthquake/Nuclear explosion? Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 28 / 35

  39. Logistics Agenda Linear Regression Logistic Regression Linear classifiers with hard threshold § Linear functions can be used to do classification as well as regression. § For example, Figure credit:AIMA: Russell, Norvig § A decision boundary is a line (or a surface, in higher dimensions) that separates the two classes. § A linear function gives rise to a linear separator and the data that adit such a separator are called linearly separable. Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 29 / 35

  40. Logistics Agenda Linear Regression Logistic Regression Linear Classifier with Hard Threshold § The linear separator in the associated fig is given by, x2= 1.7x1− 4.9 =⇒ −4.9 + 1.7x1− x2= 0   x0 x1 x2  = 0 =⇒ [−4.9,1.7,4.9] θTx = 0 Figure credit:AIMA: Russell, Norvig Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 30 / 35

  41. Logistics Agenda Linear Regression Logistic Regression Linear Classifier with Hard Threshold Figure credit:AIMA: Russell, Norvig § The explosions (y = 1) are to the right of this line with higher values of x1and lower values of x2. So, they are points for which θTx ≥ 0 § Similarly earthquakes (y = 0) are to the left of this line. So, they are points for which θTx < 0 § The classification rule is then, ( 0 if θTx ≥ 0 otherwise 1 y(x) = Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 31 / 35

  42. § To get the linear separator we have find the θ which minimizes classification error on the training set. § For regression problems, we found θ in both closed form and by gradient descent. But both approaches required us to compute the gradient. § This is not possible for the above threshold function as the gradient is undefined when the value at x − axis = 0 and 0 elsewhere. Logistics Agenda Linear Regression Logistic Regression Linear classifiers with hard threshold § Alternatively, we can think y as the result of passing the linear function θTx through a threshold function. Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 32 / 35

  43. Logistics Agenda Linear Regression Logistic Regression Linear classifiers with hard threshold § Alternatively, we can think y as the result of passing the linear function θTx through a threshold function. § To get the linear separator we have find the θ which minimizes classification error on the training set. § For regression problems, we found θ in both closed form and by gradient descent. But both approaches required us to compute the gradient. § This is not possible for the above threshold function as the gradient is undefined when the value at x − axis = 0 and 0 elsewhere. Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 32 / 35

  44. § Perceptron Learning Rule can find a linear separator given the data is linearly separable . § For data that are not linearly separable, the Perceptron algorithm fails. § So, we need to go for a gradient based optimization approach § Thus, we need to approximate hard threshold function with something smooth. 1 1 + e−u y = σ(hθ(x)) = σ(θTx) σ(u) = § Notice that the output is a number between 0 and 1, so it can be interpreted as a probability value belonging to Class 1. § This is called a logistic regression classifier. The gradient computation is tedious but straight forward. Logistics Agenda Linear Regression Logistic Regression Linear classifiers with hard threshold § Perceptron Rule - This algorithm doesnot compute the gradient to find θ. Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 33 / 35

  45. § So, we need to go for a gradient based optimization approach § Thus, we need to approximate hard threshold function with something smooth. 1 σ(u) = 1 + e−u y = σ(hθ(x)) = σ(θTx) § Notice that the output is a number between 0 and 1, so it can be interpreted as a probability value belonging to Class 1. § This is called a logistic regression classifier. The gradient computation is tedious but straight forward. Logistics Agenda Linear Regression Logistic Regression Linear classifiers with hard threshold § Perceptron Rule - This algorithm doesnot compute the gradient to find θ. § Perceptron Learning Rule can find a linear separator given the data is linearly separable . § For data that are not linearly separable, the Perceptron algorithm fails. Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 33 / 35

  46. Logistics Agenda Linear Regression Logistic Regression Linear classifiers with hard threshold § Perceptron Rule - This algorithm doesnot compute the gradient to find θ. § Perceptron Learning Rule can find a linear separator given the data is linearly separable . § For data that are not linearly separable, the Perceptron algorithm fails. § So, we need to go for a gradient based optimization approach § Thus, we need to approximate hard threshold function with something smooth. 1 1 + e−u y = σ(hθ(x)) = σ(θTx) σ(u) = § Notice that the output is a number between 0 and 1, so it can be interpreted as a probability value belonging to Class 1. § This is called a logistic regression classifier. The gradient computation is tedious but straight forward. Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 33 / 35

  47. § The joint probability of all the labels N Q § So the log likelihood for logistic regression is given by, N X ? σ?θTx(i)??y(i)? 1 − σ?θTx(i)??(1−y(i)) i=1 y(i)logσ?θTx(i)?+ (1 − y(i))log?1 − σ?θTx(i)?? l(θ) = i=1 Logistics Agenda Linear Regression Logistic Regression Maximum Likelihood Estimation of Logistic Regression § Mathematically, the probability that an example belongs to class 1 is P(y(i)= 1|x(i);θ) = σ?θTx(i)? § Thus, P(y(i)|x(i);θ) = § Similarly, P(y(i)= 0|x(i);θ) = 1 − σ?θTx(i)? ? σ?θTx(i)??y(i)? 1 − σ?θTx(i)??(1−y(i)) Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 34 / 35

  48. § So the log likelihood for logistic regression is given by, N X y(i)logσ?θTx(i)?+ (1 − y(i))log?1 − σ?θTx(i)?? l(θ) = i=1 Logistics Agenda Linear Regression Logistic Regression Maximum Likelihood Estimation of Logistic Regression § Mathematically, the probability that an example belongs to class 1 is P(y(i)= 1|x(i);θ) = σ?θTx(i)? § Thus, P(y(i)|x(i);θ) = § The joint probability of all the labels N Q § Similarly, P(y(i)= 0|x(i);θ) = 1 − σ?θTx(i)? σ?θTx(i)??y(i)? 1 − σ?θTx(i)??(1−y(i)) ? 1 − σ?θTx(i)??(1−y(i)) ? σ?θTx(i)??y(i)? i=1 Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 34 / 35

  49. Logistics Agenda Linear Regression Logistic Regression Maximum Likelihood Estimation of Logistic Regression § Mathematically, the probability that an example belongs to class 1 is P(y(i)= 1|x(i);θ) = σ?θTx(i)? § Thus, P(y(i)|x(i);θ) = § The joint probability of all the labels N Q § So the log likelihood for logistic regression is given by, N X § Similarly, P(y(i)= 0|x(i);θ) = 1 − σ?θTx(i)? σ?θTx(i)??y(i)? 1 − σ?θTx(i)??(1−y(i)) ? 1 − σ?θTx(i)??(1−y(i)) ? σ?θTx(i)??y(i)? i=1 y(i)logσ?θTx(i)?+ (1 − y(i))log?1 − σ?θTx(i)?? l(θ) = i=1 Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 34 / 35

  50. Logistics Agenda Linear Regression Logistic Regression Maximum Likelihood Estimation of Logistic Regression § Derivative of log likelihood w.r.t. one component of θ, ∂l(θ) ∂θj ∂θj i=1 X N X N X N X § This is used in an iterative gradient ascent loop. N X h h h h y(i)logσ?θTx(i)?+ (1 − y(i))log?1 − σ?θTx(i)?? y(i) σ?θTx(i)? − y(i) σ?θTx(i)? − σ?θTx(i)??1 − σ?θTx(i)?? y(i)− σ?θTx(i)?i ∂ = i∂ i σ?θTx(i)?? N ∂θjσ?θTx(i)? σ?θTx(i)?? 1 − y(i) 1 − σ?θTx(i)? 1 − y(i) 1 − σ?θTx(i)? = i=1 1 − σ?θTx(i)?? 1 − σ?θTx(i)?? x(i) j = i=1 y(i)− σ?θTx(i)? i x(i) j = i=1 x(i) j (12) = i=1 Abir Das (IIT Kharagpur) CS60010 Jan 22, 23 and 24, 2020 35 / 35

More Related