1 / 43

ICS 278: Data Mining Lectures 7, 8: Regression Algorithms

ICS 278: Data Mining Lectures 7, 8: Regression Algorithms. Padhraic Smyth Department of Information and Computer Science University of California, Irvine. Notation. Variables X, Y….. with values x, y (lower case) Vectors indicated by X Components of X indicated by X j with values x j

connor-blake
Download Presentation

ICS 278: Data Mining Lectures 7, 8: Regression Algorithms

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ICS 278: Data MiningLectures 7, 8: Regression Algorithms Padhraic Smyth Department of Information and Computer Science University of California, Irvine Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  2. Notation • Variables X, Y….. with values x, y (lower case) • Vectors indicated by X • Components of X indicated by Xj with values xj • “Matrix” data set D with n rows and p columns • jth column contains values for variable Xj • ith row contains a vector of measurements on object i, indicated by x(i) • The jth measurement value for the ith object is xj(i) • Unknown parameter for a model = q • Can also use other Greek letters, like a, b, d, g • Vector of parameters = q Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  3. Example: Multivariate Linear Regression • Task: predict real-valued Y, given real-valued vector X • Score function, e.g., S(q) = Si[y(i)– f(x(i) ; q)]2 • Model structure: f(x ; q) = a0 + Saj xj • Model parameters = q = {a0, a1, …… ap} Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  4. S = S e2 = e’ e = (y – X a)’ (y – X a) • = y’ y – a’ X’ y – y’ X a + a’ X’ X a • = y’ y – 2 a’ X’ y + a’ X’ X a • Taking derivative of S with respect to the components of a gives…. • dS/da = -2X’y + 2 X’ X a • Set this to 0 to find the extremum (minimum) of S as a function of a… • - 2X’y + 2 X’ X a = 0 • X’Xa = X’ y Letting X’X = C, and X’y = b, we have C a = b, i.e., a set of linear equations We could solve this directly by matrix inversion, i.e., a = C-1 b = ( X’ X )-1 X’ y …. but there are more numerically-stable ways to do this (e.g., LU-decomposition) Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  5. Comments on Multivariate Linear Regression • prediction is a linear function of the parameters • Score function: quadratic in predictions and parameters • Derivative of score is linear in the parameters • Leads to a linear algebra optimization problem, i.e., Ca = b • Model structure is simple…. • p-1 dimensional hyperplane in p-dimensions • Linear weights => interpretability • Useful as a baseline model • to compare more complex models to Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  6. Limitations of Linear Regression • True relationship of X and Y might be non-linear • Suggests generalizations to non-linear models • Complexity: • O(p3) - could be a problem for large p • Correlation/Collinearity among the X variables • Can cause numerical instability (C may be ill-conditioned) • Problems in interpretability (identifiability) • Includes all variables in the model… • But what if p=100 and only 3 variables are related to Y? Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  7. Finding the k best variables • Find the subset of k variables that predicts best: • This is a generic problem when p is large(arises with all types of models, not just linear regression) • Now we have models with different complexity.. • E.g., p models with a single variable • p(p-1)/2 models with 2 variables, etc… • 2p possible models in total • Note that when we add or delete a variable, the optimal weights on the other variables will change in general • k best is not the same as the best k individual variables • What does “best” mean here? • Return to this later Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  8. Search Problem • How can we search over all 2p possible models? • exhaustive search is clearly infeasible • Heuristic search is used to search over model space: • Forward search (greedy) • Backward search (greedy) • Generalizations (add or delete) • Think of operators in search space • Branch and bound techniques • This type of variable selection problem is common to many data mining algorithms • Outer loop that searches over variable combinations • Inner loop that evaluates each combination Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  9. Empirical Learning • Squared Error score (as an example: we could use other scores) S(q) = Si[y(i)– f(x(i) ; q)]2 where S(q) is defined on the training data D • We are really interested in finding the f(x; q) that best predicts y on future data, i.e., minimizing E [S] = E [y– f(x ; q)]2 • Empirical learning • Minimize S(q) on the training data Dtrain • If Dtrain is large and model is simple we are assuming that the best f on training data is also the best predictor f on future test data Dtest Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  10. Complexity versus Goodness of Fit Training data y x Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  11. Complexity versus Goodness of Fit Too simple? Training data y y x x Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  12. Complexity versus Goodness of Fit Too simple? Training data y y x x Too complex ? y x Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  13. Complexity versus Goodness of Fit Too simple? Training data y y x x Too complex ? About right ? y y x x Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  14. Complexity and Generalization Score Function e.g., squared error Stest(q) Strain(q) Complexity = degrees of freedom in the model (e.g., number of variables) Optimal model complexity Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  15. Defining what “best” means • How do we measure “best”? • Best performance on the training data? • K = p will be best (i.e., use all variables) • So this is not useful • Note: • Performance on the training data will in general be optimistic • Alternatives: • Measure performance on a single validation set • Measure performance using multiple validation sets • Cross-validation • Add a penalty term to the score function that “corrects” for optimism • E.g., “regularized” regression: SSE + l sum of weights squared Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  16. Using Validation Data Use this data to find the best q for each model fk(x ; q) Training Data • Use this data to • calculate an estimate of Sk(q) for each fk(x ; q) and • select k* = arg mink Sk(q) Validation Data Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  17. Using Validation Data Use this data to find the best q for each model fk(x ; q) Training Data • Use this data to • calculate an estimate of Sk(q) for each fk(x ; q) and • select k* = arg mink Sk(q) Validation Data Use this data to calculate an unbiased estimate of Sk*(q) for the selected model Test Data Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  18. Using Validation Data can generalize to cross-validation…. Use this data to find the best q for each model fk(x ; q) Training Data • Use this data to • calculate an estimate of Sk(q) for each fk(x ; q) and • select k* = arg mink Sk(q) Validation Data Use this data to calculate an unbiased estimate of Sk*(q) for the selected model Test Data Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  19. 2 different (but related) issues here • 1. Finding the function f that minimizes S(q) for future data • 2. Getting a good estimate of S(q), using the chosen function, on future data, • e.g., we might have selected the best function f, but our estimate of its performance will be optimistically biased if our estimate of the score uses any of the same data used to fit and select the model. Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  20. Non-linear models, linear in parameters • We can add additional polynomial terms in our equations, e.g., all “2nd order” terms f(x ; q) = a0 + Saj xj + Sbij xi xj • Note that it is a non-linear functional form, but it is linear in the parameters (so still referred to as “linear regression”) • We can just treat the xi xj termsas additional fixed inputs • In fact we can add in any non-linear input functions!, e.g. f(x ; q) = a0 + Saj fj(x) Comments: • Exact same linear algebra for optimization (same math) • Number of parameters has now exploded -> greater chance of overfitting • Ideally would like to select only the useful quadratic terms • Can generalize this idea to higher-order interactions Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  21. Non-linear (both model and parameters) • We can generalize further to models that are nonlinear in all aspects f(x ; q) = a0 + Sak gk(bk0 +Sbkj xj) where the g’s are non-linear functions with fixed functional forms. In machine learning this is called a neural network In statistics this might be referred to as a generalized linear model or projection-pursuit regression For almost any score function of interest, e.g., squared error, the score function is a non-linear function of the parameters. Closed form (analytical) solutions are rare. Thus, we have a multivariate non-linear optimization problem (which may be quite difficult!) Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  22. Optimization of a non-linear score function • We seek the minimum of a function in d dimensions, where d is the number of parameters (d could be large!) • There are a multitude of heuristic search techniques (see chapter 8) • Steepest descent (follow the gradient) • Newton methods (use 2nd derivative information) • Conjugate gradient • Line search • Stochastic search • Genetic algorithms • Two cases: • Convex (nice -> means a single global optimum) • Non-convex (multiple local optima => need multiple starts) Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  23. Other non-linear models • Splines • “patch” together different low-order polynomials over different parts of the x-space • Works well in 1 dimension, less well in higher dimensions • Memory-based models y’ = S w(x’,x) y, where y’s are from the training dataw(x’,x) = function of distance of x from x’ • Local linear regressiony’ = a0 + Saj xj , where the alpha’s are fit at prediction time just to the (y,x) pairs that are close to x’ Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  24. Time-series prediction as regression • Measurements over time x1,…… xt • We want to predict xt+1 given x1,…… xt • Autoregressive model xt+1 = f( x1,…… xt ; q ) = Sak xt-k • Number of coefficients K = memory of the model • Can take advantage of regression techniques in general to solve this problem (e.g., linear in parameters, score function = squared error, etc) • Generalizations • Vector x • Non-linear function instead of linear • Add in terms for time-trend (linear, seasonal), for “jumps”, etc Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  25. Probabilistic Interpretation of Regression y = f(x) + e • e = noise model, e.g., zero-mean additive Gaussian • If e is zero mean, then E[y] = f(x) • Expectation is with respect to p(y|x) • Key idea is that the observed y is viewed as a stochastic function of x • Uncertainty (reflected in noise term e) comes from both measurement noise and unobserved variables that affect y • Learning • Empirical sum of squares, S ( y(i) – f[ x(i) ] )2, is minimized (asymptotically) by setting f[ x(i) = E[y] • i.e., we are really trying to learn expected value of y as a function of x • Probabilistic framework allows for a broad range of generalizations • Different error models (non-additive, non-constant) • Bayesian regression (priors on parameters, etc) • etc Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  26. Other aspects of regression • Diagnostics • Useful in low dimensions • Weighted regression • Useful when rows have different weights • Different score functions • E.g. absolute error, or additive noise varies as a function of x • Predicting y values constrained to a certain range, e.g., y > 0, or 0 < y < 1 • Predicting binary y values • Regression as a generalization of classification Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  27. Classification as a special case of Regression • Let y take values 0 or 1 • Under certain score functions (squared error, certain types of likelihood), regression will try to learn an f function that matches E[y] • For binary y we have E[y] = 1. p(y=1|x) + 0 p(y=0|x) = p(y=1|x) • Thus, if regression drives f towards E[y], then for binary classification problems a regression model will try to approximate p(y=1|x) (posterior class probabilities) • e.g., this is what logistic regression and neural networks do Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  28. Predicting an output between 0 and 1 • We often have a problem where y lies between 0 and 1 • probability that a person with attributes X will survive 10 years • proportion of people in Zip code X who will buy a product • We could use linear regression, but….. • Instead we can use the logistic function log p(y=1|x)/log p(y=0|x) = a0 + Saj xj Equivalently, p(y=1|x) = 1/[1 + exp(- a0 - Saj xj ) ] We model the log-odds as a linear function of the input variables. This is known as logistic regression. Could be viewed as a simple neural network with a single hidden unit Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  29. Logistic Regression • How do we estimate the alpha parameters? • Problem: linear algebra solution no longer applies since our function form (and score) is now non-linear in the parameters • Common approach • Score function = likelihood = probability of observed data • Select parameters to maximize the log-likelihood (“maximum likelihood”) • S(q) = Si log p( y(i) | x(i) ; a)= Si y(i) log p( y(i)=1| x(i) ; a) + [1-y(i)] log(1- p( y(i)=1| x(i) ; a)) • Can compute the 2nd derivative directly as weighted matrix • Forms the basis for an iterative 2nd order Newton scheme • Each iteration solves a weighted regression problem, O(p3) • But see (e.g.) Komarek and Moore (2005) for speedups for sparse data • Known as iteratively reweighted least-squares • Log-likelihood here is convex: so it is quite stable (only one global maximum!). Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  30. Generalized Linear Models (GLMs)(McCullagh and Nelder, 1989) • g(y) = u(x) = a0 + Saj xj • Where g [ ] is a “link” function • u(x) is a linear function of the vector x • Examples: • g = identity function -> linear regression • Logistic regression: g(y) = log(y / 1-y) = a0 + Saj xj • Logarithmic link: g(y) = log(y) = a0 + Saj xj • GLMs are widely used in statistics • Details of learning/fitting algorithm depend on the specifics of the link function Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  31. Tree-Structured Regression • Functional form of model is a “regression tree” • Univariate thresholds at internal nodes • Constant or linear surfaces at the leaf nodes • Yields piecewise constant (or linear) surface • (like classification tree, but for regression) • Very crude functional form…. but • Can be very useful in high-dimensional problems • Can useful for interpretation • Can handle combinations of real and categorical variables • Search problem • Finding the optimal tree is NP-hard (for any reasonable score) • Practice: greedy algorithms Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  32. More on regression trees • Greedy search algorithm • For each variable, find the best split point such the mean of Y either side of the split minimizes the mean-squared error • Select the variable with the minimum average error • Partition the data using the threshold • Recursively apply this selection procedure to each “branch” • What size tree? • A full tree will likely overfit the data • Common methods for tree selection • Grow a large tree and select an appropriate subtree by Xvalidation • Grow a number of small fixed-sized trees and average their predictions • Will discuss trees further in lectures on classification: very similar ideas used for building classification trees and regression trees Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  33. Model Averaging • Can average over parameters and models • E.g., weighted linear combination of predictions from multiple models y = S wk yk • Why? Any predictions from a point estimate of parameters or a single model has only a small chance of the being the best • Averaging makes our predictions more stable and less sensitive to random variations in a particular data set (good for less stable models like trees) Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  34. Model Averaging • Model averaging flavors • Fully Bayesian: average over uncertainty in parameters and models • “empirical Bayesian”: learn weights over multiple models • E.g., stacking and bagging • Build multiple simple models in a systematic way and combine them, e.g., • Boosting: will say more about this in later lectures • Random forests: stochastically perturb the data, learn multiple trees, and then combine for prediction Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  35. Components of Data Mining Algorithms • Representation: • Determining the nature and structure of the representation to be used; • Score function • quantifying and comparing how well different representations fit the data • Search/Optimization method • Choosing an algorithmic process to optimize the score function; and • Data Management • Deciding what principles of data management are required to implement the algorithms efficiently. Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  36. What’s in a Data Mining Algorithm? Task Representation Score Function Search/Optimization Data Management Models, Parameters Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  37. Multivariate Linear Regression Task Regression Y = Weighted linear sum of X’s Representation Score Function Least-squares Search/Optimization Linear algebra Data Management None specified Models, Parameters Regression coefficients Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  38. Autoregressive Time Series Models Task Time Series Regression X = Weighted linear sum of earlier X’s Representation Score Function Least-squares Search/Optimization Linear algebra Data Management None specified Models, Parameters Regression coefficients Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  39. Neural Networks Task Regression Representation Y = nonlin function of X’s Score Function Least-squares Search/Optimization Gradient descent Data Management None specified Models, Parameters Network weights Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  40. Logistic Regression Task Regression Log-odds(Y) = linear function of X’s Representation Score Function Log-likelihood Search/Optimization Iterative (Newton) method Data Management None specified Models, Parameters Logistic weights Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  41. Software • MATLAB • Many free “toolboxes” on the Web for regression and prediction • e.g., see http://lib.stat.cmu.edu/matlab/ and in particular the CompStats toolbox • R • General purpose statistical computing environment (successor to S) • Free (!) • Widely used by statisticians, has a huge library of functions and visualization tools • Commercial tools • SAS, Salford Systems, other statistical packages • Various data mining packages • Often are not progammable: offer a fixed menu of items Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  42. Suggested Reading in Text • Chapter 4: • General statistical aspects of model fitting • Pages 93 to 116, plus Section 4.7 on sampling • Chapter 5: • “reductionist” view of learning algorithms (can skim this) • Chapter 6: • Different forms of functional forms for modeling • Pages 165 to 183 • Chapter 8: • Section 8.3 on multivariate optimization • Chapter 9: • linear regression and related methods • Can skip Section 11.3 Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

  43. Useful References N. R. Draper and H. Smith, Applied Regression Analysis, 2nd edition, Wiley, 1981 (the “bible” for classical regression methods in statistics) T. Hastie, R. Tibshirani, and J. Friedman, Elements of Statistical Learning, Springer Verlag, 2001 (statistically-oriented overview of modern ideas in regression and classificatio, mixes machine learning and statistics) Data Mining Lectures Lecture 7/8: Regression Padhraic Smyth, UC Irvine

More Related