Basis Expansion and Regularization

1. Basis Expansion and Regularization Prof.Liqing Zhang Dept. Computer Science & Engineering, Shanghai Jiaotong University

2. Outline • Piece-wise Polynomials and Splines • Wavelet Smoothing • Smoothing Splines • Automatic Selection of the Smoothing Parameters • Nonparametric Logistic Regression • Multidimensional Splines • Regularization and Reproducing Kernel Hilbert Spaces Basis Expansion and Regularization

3. Piece-wise Polynomials and Splines • Linear basis expansion • Some basis functions that are widely used Basis Expansion and Regularization

4. Regularization • Three approaches for controlling the complexity of the model. • Restriction • Selection • Regularization: Basis Expansion and Regularization

5. Piecewise Polynomials and Splines Basis Expansion and Regularization

6. Piecewise Cubic Polynomials • Increasing orders of continuity at the knots. • A cubic spline with knots at and : • Cubic spline truncated power basis Basis Expansion and Regularization

7. Piecewise Cubic Polynomials • An order-M spline with knots , j=1,…,K is a piecewise-polynomial of order M, and has continuous derivatives up to order M-2. • A cubic spline has M=4. • Truncated power basis set: Basis Expansion and Regularization

8. Natural boundary constraints Linear Cubic Cubic Linear Natural cubic spline • Natural cubic spline adds additional constraints, namely that the function is linear beyond the boundary knots. Basis Expansion and Regularization

9. B-spline • The augmented knot sequenceτ: • Bi,m(x), the i-th B-spline basis function of order m for the knot-sequence τ, m≤M. Basis Expansion and Regularization

10. The sequence of B-spline up to order 4 with ten knots evenly spaced from 0 to 1 The B-spline have local support; they are nonzero on an interval spanned by M+1 knots. B-spline Basis Expansion and Regularization

11. Smoothing Splines • Base on the spline basis method: • So , is the noise. • Minimize the penalized residual sum of squares is a fixed smoothing parameter f can be any function that interpolates the data the simple least squares line fit Basis Expansion and Regularization

12. Smoothing Splines • The solution is a natural spline: • Then the criterion reduces to: • where • So the solution: • The fitted smoothing spline: Basis Expansion and Regularization

13. Smoothing Splines • Function of age, that response the relative change in bone mineral density measured at the spline in adolescents • Separate smoothing splines fit the males and females, • 12 degrees of freedom 脊骨BMD—骨质密度 Basis Expansion and Regularization

14. Smoothing Matrix • the N-vector of fitted values • The finite linear operator — the smoother matrix • Compare with the linear operator in the LS-fitting: M cubic-spline basis functions, knot sequenceξ • Similarities and differences: • Both are symmetric, positive semidefinite matrices • idempotent（幂等的) ; shrinking • rank: Basis Expansion and Regularization

15. Smoothing Matrix • Effective degrees of freedom of a smoothing spline • in the Reinsch form: • Since , solution: • is symmetric and has a real eigen-decomposition • is the corresponding eigenvalue of K Basis Expansion and Regularization

16. Smoothing spline fit of ozone(臭氧) concentration versus Daggot pressure gradient. • Smoothing parameter df=5 and df=10. • The 3rd to 6th eigenvectors of the spline smoothing matrices Basis Expansion and Regularization

17. The smoother matrix for a smoothing spline is nearly banded, indicating an equivalent kernel with local support. Basis Expansion and Regularization

18. Bias-Variance Tradeoff • Example: • For • The diagonal contains the pointwise variances at the training • Bias is given by • is the (unknown) vector of evaluations of the true f Basis Expansion and Regularization

19. df=9, bias slight, variance not increased appreciably • df=15, over learning, standard error widen Bias-Variance Tradeoff • df=5, bias high, standard error band narrow Basis Expansion and Regularization

20. The EPE and CV curves have the a similar shape. And, overall the CV curve is approximately unbiased as an estimate of the EPE curve Bias-Variance Tradeoff • The integrated squared prediction error (EPE) combines both bias and variance in a single summary: • N fold (leave one) cross-validation: Basis Expansion and Regularization

21. Logistic Regression • Logistic regression with a single quantitative input X • The penalized log-likelihood criterion Basis Expansion and Regularization

22. Multidimensional Splines • Tensor product basis • The M1×M2 dimensional tensor product basis • , basis function for coordinate X1 • , basis function for coordinate X2 Basis Expansion and Regularization

23. Tenor product basis of B-splines, some selected pairs Basis Expansion and Regularization

24. Multidimensional Splines • High dimension smoothing Splines • J is an appropriate penalty function a smooth two-dimensional surface, a thin-plate spline. • The solution has the form Basis Expansion and Regularization

25. Multidimensional Splines • The decision boundary of an additive logistic regression model. Using natural splines in each of two coordinates. • df = 1 +(4-1) + (4-1) = 7 Basis Expansion and Regularization

26. Multidimensional Splines • The results of using a tensor product of natural spline basis in each coordinate. • df = 4 x 4 = 16 Basis Expansion and Regularization

27. Multidimensional Splines • A thin-plate spline fit to the heart disease data. • The data points are indicated, as well as the lattice of points used as knots. Basis Expansion and Regularization

28. Reproducing Kernel Hilbert space • A regularization problems has the form: • L(y,f(x)) is a loss-function. • J(f) is a penalty functional, and H is a space of functions on which J(f) is defined. • The solution • span the null space of the penalty functional J Basis Expansion and Regularization

29. Spaces of Functions Generated by Kernel • Important subclass are generated by the positive kernel K(x,y). • The corresponding space of functions Hk is called reproducing kernel Hilbert space. • Suppose that K has an eigen-expansion • Elements of H have an expansion Basis Expansion and Regularization

30. Spaces of Functions Generated by Kernel • The regularization problem become • The finite-dimension solution(Wahba,1990) • Reproducing properties of kernel function Basis Expansion and Regularization

31. Spaces of Functions Generated by Kernel • The penalty functional • The regularization function reduces to a finite-dimensional criterion • K is NxN matrix, the ij-th entry K(xi, xj) Basis Expansion and Regularization

32. RKHS • Penalized least squares • The solution of : • The fitted values: • The vector of N fitted value is given by Basis Expansion and Regularization

33. Example of RKHS • Polynomial regression • Suppose M huge • Given • Loss function: • The penalty polynomial regression: Basis Expansion and Regularization

34. Penalized Polynomial Regression • Kernel: has eigen-functions • E.g. d=2, p=2: • The penalty polynomial regression: Basis Expansion and Regularization

35. RBF kernel & SVM kernel • Gaussian Radial Basis Functions • Support Vector Machines Basis Expansion and Regularization

36. Wavelet smoothing • Another type of bases——Wavelet bases • Wavelet bases are generated by translations and dilations of a single scaling function . • If ,then generates an orthonormal basis for functions with jumps at the integers. • form a space called reference space V0 • The dilations form an orthonormal basis for a space V1 V0 • Generally, we have Basis Expansion and Regularization

37. Basis Expansion and Regularization

38. Basis Expansion and Regularization

39. Wavelet smoothing • The L2 space dividing • Mother wavelet generate function form an orthonormal basis for W0. Likewise form a basis for Wj. Basis Expansion and Regularization

40. Wavelet smoothing • Wavelet basis on W0 : • Wavelet basis on Wj : • The symmlet-p wavelet: • A support of 2p-1 consecutive intervals. • p vanishing moments: Basis Expansion and Regularization

41. Basis Expansion and Regularization

42. Basis Expansion and Regularization

43. Basis Expansion and Regularization

44. Adaptive Wavelet Filtering • Wavelet transform: • y: response vector, W: NxN orthonormal wavelet basis matrix • Stein Unbiased Risk Estimation (SURE) • The solution: • Fitted function is given by inverse wavelet transform: , LS coefficients truncated to 0 • Simple choice for Basis Expansion and Regularization

45. Basis Expansion and Regularization

46. Basis Expansion and Regularization

47. Ex. 5.9 Derive the Reinsch form for the smoothing spline. Ex. 5.16 Consider the ridge regression problem (5.53), and assume . Assume you have a kernel K that computes the inner product • Derive (5.62) on page 171 in the text. How would you compute the matrices V and , given K? Hence show that (5.63) is equivalent to (5.53). • Show that (5.75) where H is the N×M matrix of evaluations , and the N×N matrix of inner-products . Basis Expansion and Regularization