Review of Linear Algebra

1. Review of Linear Algebra Fall 2014 The University of Iowa Tianbao Yang

2. Announcements • TA: Shiyao Wang • Office hours: 3:30-5:00 pm Tu/Th • Office Location: 201C • Homework-1 is available on ICON

3. Today’s Topics • Vectorand Matrix • Operation on Matrices/Vectors • Singular value decomposition • Norms • An Application in Text Analysis

4. Vector • Scalar • a real number: 7 • Vector • one dimensional array • representation: column vector • representation: row vector

5. Vector • Dimensionality or size: • number of scalars • Vector Space • all vectors of the same dimension

6. Matrix • Two dimensional array • Representation • (i,j)-th element: • A set of vectors • vector: a special matrix rows columns

7. Matrix • Dimensionality or size • m*n (m rows and n columns) • Matrix Space:

8. Today’s Topics • Vectorand Matrix • Operation on Matrices/Vectors • Singular value decomposition • Norms • An Application in Text Analysis

9. Operations • Matrix addition: • two matrices of the same size • (i,j)-th element: • Scalar multiplication: • results in the same size • Matrix subtraction:

10. Operations • Multiplication of a row vector and a column vector • Matrix Multiplication • ,

11. Operations

12. Operations • Transpose: • (i,j)-the element: • transpose of a column vector: row vector • Rules:

13. Special Matrices • Square matrix: • Symmetric matrix: • Zero matrix • all elements are zeros • Identity Matrix: • each column (or row) standard basis • :

14. Operations • (Square) Matrix Inverse • similar to inverse of a scalar: • inverse of a square matrix: • if there exists: Non-singular

15. Operations • Trace of a square matrix: • definition • rules

16. Today’s Topics • Vectorand Matrix • Operation on Matrices/Vectors • Singular value decomposition • Norms • An Application in Text Analysis

17. mm mm mn mn V is nn V is nn Singular Value Decomposition • A matrix: • Singular Value Decomposition (SVD) • The columns of are left singular vectors • The columns of are right singular vectors • is a diagonal matrix with singular values (positive values)

18. mm mn V is nn Singular Value Decomposition • Illustration of SVD dimensions and sparseness

19. Singular Value Decomposition • Rank of a Matrix • organize singular values in descending order • the largest index that is non-zero

20. Eigen-value Decomposition • Eigenvectors(for a square mm matrix S) • Example (right) eigenvector eigenvalue

21. Eigen-value Decomposition

22. Eigen-value Decomposition S = U *  * UT

23. Eigen-value Decomposition S = U *  * UT

24. Eigen-value Decomposition • This is generally true for symmetric square matrix • Columns of U are eigenvectors of S • Diagonal elements of  are eigenvalues of S S = U *  * UT

25. mm mn V is nn nn nn nn Eigen-value Decomposition • A symmetric matrix: • Eigen-value Decomposition • The columns of are eigen-vectors • is a diagonal matrix with real eigen-values

26. mm mn V is nn nn nn nn Positive (Semi-)Definite Matrix • A symmetric matrix: • Eigen-value Decomposition • The columns of are eigen-vectors • is a diagonal matrix with Positive eigen-values • is a diagonal matrix with Non-negative eigen-values

27. Today’s Topics • Vectorand Matrix • Operation on Matrices/Vectors • Singular value decomposition • Norms • An Application in Text Analysis

28. Inner Product • inner product between two vectors • Norm of a Vector: (Euclidean Norm, norm)

29. Inequalities • Cauchy-Schwarz Inequality • Triangle Inequality

30. p-Norm of a Vector • p-norm • p = 1 norm • p = 2 norm • p =  norm

31. Norm of a Matrix • Inner Product between two matrices • Norm of a Matrix (Frobenius norm)

32. Other Matrix Norms • Induced Norm (operator norm): • p=2, spectral norm: maximum singular value • p=1, maximum absolute column sum • p= , maximum absolute row sum

33. Other Matrix Norms • Schatten Norm: • p=1, trace norm (or nuclear norm) • p=2, Frobenius norm • p= , Spectral norm

34. Machine Learning Problems • Solve the following problems Loss norm

35. Today’s Topics • Vectorand Matrix • Operation on Matrices/Vectors • Singular value decomposition • Norms • An Application in Search Engine

36. Search Engine • A database of Webpages • A user-typed query • generate a list of relevant webpages • A ranking problem https://www.facebook.com/ contain query words (LSI) a lot of links to them (PageRank)

37. Representation of documents • webpage is a document • document contains many terms (words) • To represent a document • collect all meaningful terms • count the occurrence of each term in a document

38. Representation of documents • Term-Document Matrix

39. Search Engine • Represent the query in the same way • e.g. query: “computer system” Query 0 0 1 0 1 0 0 0 0 0 0 0

40. Search Engine • Retrieve Similar Documents • Query • Similarity • inner product • normalized inner product (cosine similarity) • Assume A is column normalized and q is normalized

41. Concept Concept Rep. of Concepts in term space Rep. of concepts in document space Search Engine • Latent Semantic Indexing • SVD

42. Search Engine • Low rank approximation: • approximate matrix with the largest singular values and singular vectors Rank-k approximation

43. Search Engine • Why Low rank approximation: • data compression: billions to thousands • filter out noise Rank-k approximation

44. Finding “Good Concepts”

45. X X SVD: Example: m=2

46. X X LSI: Example: m=2

47. X X LSI: Example: m=2

48. X X LSI: Example: m=2

49. LSI: Example: m=3 Top three left singular vectors -0.2214 -0.1132 0.2890 -0.1976 -0.0721 0.1350 -0.2405 0.0432 -0.1644 -0.4036 0.0571 -0.3378 -0.6445 -0.1673 0.3611 -0.2650 0.1072 -0.4260 -0.2650 0.1072 -0.4260 -0.3008 -0.1413 0.3303 -0.2059 0.2736 -0.1776 -0.0127 0.4902 0.2311 -0.0361 0.6228 0.2231 -0.0318 0.4505 0.1411

50. Search Engine • Why Low rank approximation: • data compression: billions to thousands • filter out noise