1 / 12

CS 395/495-26: Spring 2004

CS 395/495-26: Spring 2004. IBMR: Singular Value Decomposition (SVD Review) Jack Tumblin jet@cs.northwestern.edu. Matrix Multiply: A Change-of-Axes. Matrix Multiply: Ax = b x and b are column vectors A has m rows, n columns. a 11 … a 1n … … a m1 … a mn. x 1 … x n. b 1 … b m.

dewitt
Download Presentation

CS 395/495-26: Spring 2004

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CS 395/495-26: Spring 2004 IBMR: Singular Value Decomposition(SVD Review) Jack Tumblin jet@cs.northwestern.edu

  2. Matrix Multiply: A Change-of-Axes • Matrix Multiply: Ax = b • x and b are column vectors • A has m rows, n columns a11 … a1n … … am1 … amn x1 … xn b1 … bm = b2 x2 x b Ax=b b1 b3 x1 Input (N dim) (2D) Output (M-dim) (3D)

  3. Matrix Multiply: A Change-of-Axes • Matrix Multiply: Ax = b • x and b are column vectors • A has m rows, n columns • Matrix multiply is just a set of dot-products;it ‘changes coordinates’ for vector x, makes b. • Rows of A = A1,A2,A3,... = new coordinate axes • Ax = a dot product for each new axis a11 … a1n … … am1 … amn x1 … xn b1 … bm = A2 b2 x2 x b Ax=b b1 A1 A3 b3 x1 Input (N dim) (2D) Output (M-dim) (3D)

  4. How Does Matrix ‘stretch space’? • Sphere of all unit-length x ellipsoid of b • Output ellipsoid’s axes are always perpendicular (true for any ellipsoid) **THUS** we can make ┴, unit-length axes… ellipsoid (disk) b2 x2 b x Ax=b b1 1 1 x1 b3 u2 u1 Input (N dim.) Output (M dim.)

  5. SVD finds: ‘Output’ Vectors U, and… • Sphere of all unit-length x ellipsoid of b • Output ellipsoid’s axes form orthonormal basis vectors Ui: • Basis vectors Ui are columns of U matrix SVD(A) = USVTcolumns of U = OUTPUT basis vectors ellipsoid (disk) b2 x2 b x Ax=b b1 1 1 x1 b3 u2 u1 Input (N dim.) Output (M dim.)

  6. SVD finds: …‘Input’ Vectors V, and… • For each Ui, make a matching Vi that: • Transforms to Ui (with scaling si) : si (A Vi) = Ui • Forms an orthonormal basis of input space SVD(A) = USVTcolumns of V = INPUT basis vectors b2 x2 x b v1 Ax=b v2 b1 x1 b3 u2 u1 Input (N dim.) Output (M dim.)

  7. SVD finds: …‘Scale’ factors S. • We have Unit-length Output Ui, vectors, • Each from a Unit-length Input Vi vector, so • So we need ‘scale factor’ (singular values)si to link input to output: si (A Vi) = Ui SVD(A) = USVT s1 b2 x2 x b v1 Ax=b v2 b1 x1 b3 s2 u2 u1 Input (N dim.) Output (M dim.)

  8. SVD Review: What is it? • Finish: SVD(A) = USVT • add ‘missing’ Ui or Vi, define these si=0. • Singular matrix S: diagonals are si : ‘v-to-u scale factors’ • Matrix U, Matrix V have columns Ui and Vi b2 x2 x b v1 u3 Ax=b v2 b1 x1 b3 u2 u1 Input (N dim.) Output (M dim.) A = USVT ‘Find Input & Output Axes, Linked by Scale’

  9. ‘Let SVDs Explain it All for You’ • cool! U and V are Orthonormal! U-1 = UT, V-1 = VT • Rank(A)? == # of non-zero singular values si • ‘ill conditioned’? Some si are nearly zero • ‘Invert’ a non-square matrix A ? Amazing! pseudo-inverse: A=USVT; A VS-1UT = I; so A-1 =VS-1 UT b2 x2 x v1 u3 Ax=b v2 b1 x1 b3 u2 u1 Input (N dim.) Output (M dim.) A = USVT ‘Find Input & Output Axes, Linked by Scale’

  10. ‘Solve for the Null Space’ Ax=0 • Easy! x is the Vi axis that doesn’t affect the output (si = 0) • are allsi nonzero? then the only answer is x=0. • Null ‘space’ because Vi is a DIMENSION--- x = Vi * a • More than 1 zero-valued si? null space >1 dimension... x = aVi1 + bVi2 + … b2 x2 x v1 u3 Ax=b v2 b1 x1 b3 u2 u1 Input (N dim.) Output (M dim.) A = USVT ‘Find Input & Output Axes, Linked by Scale’

  11. More on SVDs: Handout • Zisserman Appendix 3, pg 556 (cryptic) • Linear Algebra books • Good online introduction: (your handout)Leach, S. “Singular Value Decomposition – A primer,” Unpublished Manuscript, Department of Computer Science, Brown University. http://www.cs.brown.edu/research/ai/dynamics/tutorial/Postscript/ • Google on Singular Value Decomposition...

  12. END

More Related