Basis Expansions and Generalized Additive Models (2)

1. Basis Expansions and Generalized Additive Models (2) Splines Generalized Additive Model MARS

2. Piecewise Polynomials Lower-right: Cubicspline

3. Spline • An order-M spline with knots ξj, j = 1,...,K is a piecewise-polynomial of order M, and has continuous derivatives up to order M − 2. • Cubic spline is order 4; • piecewise-constant function an order-1 spline • Basis functions: • In practice the most widely used orders are M = 1, 2 and 4.

4. Natural Cubic Splines • polynomials fit to data tends to be erratic near the boundaries, and extrapolation can be dangerous. • A natural cubic spline adds additional constraints - the function is linear beyond the boundary knots. • This requires extra constraints: the second derivative to be zero at the first and the last point Right: logistic regression.

5. Natural Cubic Splines

6. Natural Cubic Splines • How many knots, and where to place them ?? • Evenly spaced knots over the range of data (or percentiles) • Examining the fit visually • Can select the best knot setting by cross-validations.

7. Example: Logistic regression on South African Heart Disease Data X1 representing sbp, h1(X1) is a basis consisting of four basis functions…… Backward stepwise deletion process while preserving the group structure of each term

8. Smoothing Splines • Avoids the knot selection problem completely. • Uses a maximal set of knots. • The complexity of the fit is controlled by regularization. • Setup: among all functions f(x) with two continuous derivatives, find one that minimizes the penalized residual sum of squares • Lambda: smoothing parameter. • The second term penalizes curvature in the function

9. Smoothing Splines • The solution is a natural cubic spline with knots at the unique values of the xi, i = 1,...,N • the penalty term translates to a penalty on the spline coefficients  shrink toward the linear fit

10. Smoothing Splines

11. Smoothing Splines effective degrees of freedom of a smoothing spline:

12. Smoothing Splines Bias-variance trade-off

13. Multidimensional Splines • Basis of functions h1k(X1), • k = 1,...,M1 for X1 • Basis of functions h2k(X2), • k = 1,...,M2 for X2

14. Multidimensional Splines • The coefficients can be fit by least squares, as before. • But the dimension of the basis grows exponentially fast.

15. Generalized Additive Models • fi() are unspecified smooth functions • If model each function using an expansion of basis functions, the model could be fit by regression. • g(μ) = μ identity link, used for linear and additive models for Gaussian response data. • g(μ) = logit(μ) as above, or g(μ) = probit(μ), for modeling binomial probabilities. • g(μ) = log(μ) for log-linear or log-additive models for Poisson count data.

16. Generalized Additive Models • The penalized least squares: • where the λj ≥0 are tuning parameters • The minimizer is an additive cubic spline model • Each fj is a cubic spline in the component Xj, with knots at each of the unique values of xij, i = 1,...,N. • To make solution unique,

17. Generalized Additive Models • Equivalent to multiple regression for linear models: • > Can use other univariate regression smoothers such as local polynomial regression and kernel methods as Sj Current residual against all other variables Sj represents the spline Making the average zero

18. Multidimensional Splines

19. MARS: Multivariate Adaptive Regression Splines • an adaptive procedure for regression, well suited for high-dimensional problems • MARS uses expansions in piecewise linear basis functions of the form “a reflected pair”

20. MARS: Multivariate Adaptive Regression Splines • The idea is to form reflected pairs for each input Xj with knots at each observed value xij of that input. • The collection of basis functions: • If all of the input values are distinct, there are 2Np basis functions altogether. • Model: • where each hm(X) is a function in C, or a product of two or more such functions.

21. MARS: Multivariate Adaptive Regression Splines • Model building – forward stepwise: in each iteration, select a function from the set C or their products . • coefficients βm are estimated by standard linear regression. • Add terms in the form:

22. MARS: Multivariate Adaptive Regression Splines In model Candidates At each stage we consider all products of a candidate pair with a basis function in the model. The product that decreases the residual error the most is added into the current model.

23. MARS: Multivariate Adaptive Regression Splines

24. MARS: Multivariate Adaptive Regression Splines • At the end of this process we have a large model that typically overfits the data. • A backward deletion procedure is applied. • Remove the term whose removal causes the smallest increase in residual squared error, one at a time. • This produces the best model of each size (number of terms) λ. • Use (generalized) cross-validation to compare the models and select the best λ. • A link function can be used for discrete or other types of outcomes

25. MARS: Multivariate Adaptive Regression Splines Computers and Geotechnics 48 (2013) 82–95

26. MARS: Multivariate Adaptive Regression Splines