1 / 39

Linear Algebra Review (with a Small Dose of Optimization)

Linear Algebra Review (with a Small Dose of Optimization). Hristo Paskov CS246. Outline. Basic definitions Subspaces and Dimensionality Matrix functions: inverses and eigenvalue decompositions Convex optimization. Vectors and Matrices. Vector May also write. Vectors and Matrices.

Download Presentation

Linear Algebra Review (with a Small Dose of Optimization)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Linear Algebra Review(with a Small Dose of Optimization) HristoPaskov CS246

  2. Outline • Basic definitions • Subspaces and Dimensionality • Matrix functions: inverses and eigenvalue decompositions • Convex optimization

  3. Vectors and Matrices • Vector • May also write

  4. Vectors and Matrices • Matrix • Written in terms of rows or columns

  5. Multiplication • Vector-vector: • Matrix-vector:

  6. Multiplication • Matrix-matrix: =

  7. Multiplication • Matrix-matrix: • rows of , cols of

  8. Multiplication Properties • Associative • Distributive • NOT commutative • Dimensions may not even be conformable

  9. Useful Matrices • Identity matrix • Diagonal matrix

  10. Useful Matrices • Symmetric • Orthogonal • Columns/ rows are orthonormal • Positive semidefinite: • Equivalently, there exists

  11. Outline • Basic definitions • Subspaces and Dimensionality • Matrix functions: inverses and eigenvalue decompositions • Convex optimization

  12. Norms • Quantify “size” of a vector • Given , a norm satisfies • Common norms: • Euclidean -norm: • -norm: • -norm:

  13. Linear Subspaces

  14. Linear Subspaces • Subspace satisfies • If and , then • Vectors span if

  15. Linear Independence and Dimension • Vectors are linearlyindependent if • Every linear combination of the is unique • if span and are linearly independent • If span then • If then are NOT linearly independent

  16. Linear Independence and Dimension

  17. Matrix Subspaces • Matrix defines two subspaces • Column space • Row space • Nullspace of : • Analog for column space

  18. Matrix Rank • gives dimensionality of row and column spaces • If has rank , can decompose into product of and matrices =

  19. Properties of Rank • For • has fullrank if • If rows not linearly independent • Same for columns if

  20. Outline • Basic definitions • Subspaces and Dimensionality • Matrix functions: inverses and eigenvalue decompositions • Convex optimization

  21. Matrix Inverse • is invertible iff • Inverse is unique and satisfies • If is invertible then is invertible and

  22. Systems of Equations • Given wish to solve • Exists only if • Possibly infinite number of solutions • If is invertible then • Notational device, do notactually invert matrices • Computationally, use solving routines like Gaussian elimination

  23. Systems of Equations • What if ? • Find that gives closest to • is projection of onto • Also known as regression • Assume Invertible Projection matrix

  24. Systems of Equations

  25. Eigenvalue Decomposition • Eigenvalue decomposition of symmetric is • contains eigenvalues of • is orthogonal and contains eigenvectors of • If is not symmetric but diagonalizable • is diagonal by possibly complex • not necessarily orthogonal

  26. Characterizations of Eigenvalues • Traditional formulation • Leads to characteristic polynomial • Rayleigh quotient (symmetric )

  27. Eigenvalue Properties • For with eigenvalues • When is symmetric • Eigenvalue decomposition is singular value decomposition • Eigenvectors for nonzero eigenvalues give orthogonal basis for

  28. Simple Eigenvalue Proof • Why ? • Assume is symmetric and full rank • If , eigenvalue of is 0 • Since is product of eigenvalues, one of the terms is , so product is 0

  29. Outline • Basic definitions • Subspaces and Dimensionality • Matrix functions: inverses and eigenvalue decompositions • Convex optimization

  30. Convex Optimization • Find minimum of a function subject to solution constraints • Business/economics/ game theory • Resource allocation • Optimal planning and strategies • Statistics and Machine Learning • All forms of regression and classification • Unsupervised learning • Control theory • Keeping planes in the air!

  31. Convex Sets • A set is convex if and • Line segment between points in also lies in • Ex • Intersection of halfspaces • balls • Intersection of convex sets

  32. Convex Functions • A real-valued function is convex if is convex and and • Graph of upper bounded by line segment between points on graph

  33. Gradients • Differentiable convex with • Gradient at gives linear approximation

  34. Gradients • Differentiable convex with • Gradient at gives linear approximation

  35. Gradient Descent • To minimize move down gradient • But not too far! • Optimum when • Given , learning rate , starting point Do until

  36. Stochastic Gradient Descent • Many learning problems have extra structure • Computing gradient requires iterating over all points, can be too costly • Instead, compute gradient at single training example

  37. Stochastic Gradient Descent • Given , learning rate , starting point Do until nearly optimal For in random order • Finds nearly optimal

  38. Minimize

  39. Learning Parameter

More Related