Review of linear algebra
This presentation is the property of its rightful owner.
Sponsored Links
1 / 50

Review of Linear Algebra PowerPoint PPT Presentation


  • 68 Views
  • Uploaded on
  • Presentation posted in: General

Review of Linear Algebra. Fall 2014 The University of Iowa Tianbao Yang. Announcements. TA: Shiyao Wang Office hours: 3:30- 5:00 pm Tu / Th Office Location: 201C Homework-1 is available on ICON. Today’s Topics. Vector and Matrix Operation on Matrices/Vectors

Download Presentation

Review of Linear Algebra

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Review of linear algebra

Review of Linear Algebra

Fall 2014

The University of Iowa

Tianbao Yang


Announcements

Announcements

  • TA: Shiyao Wang

  • Office hours: 3:30-5:00 pm Tu/Th

  • Office Location: 201C

  • Homework-1 is available on ICON


Today s topics

Today’s Topics

  • Vectorand Matrix

  • Operation on Matrices/Vectors

  • Singular value decomposition

  • Norms

  • An Application in Text Analysis


Vector

Vector

  • Scalar

    • a real number: 7

  • Vector

    • one dimensional array

    • representation: column vector

    • representation: row vector


Vector1

Vector

  • Dimensionality or size:

    • number of scalars

  • Vector Space

    • all vectors of the same dimension


Matrix

Matrix

  • Two dimensional array

  • Representation

  • (i,j)-th element:

  • A set of vectors

  • vector: a special matrix

rows

columns


Matrix1

Matrix

  • Dimensionality or size

    • m*n (m rows and n columns)

  • Matrix Space:


Today s topics1

Today’s Topics

  • Vectorand Matrix

  • Operation on Matrices/Vectors

  • Singular value decomposition

  • Norms

  • An Application in Text Analysis


Operations

Operations

  • Matrix addition:

    • two matrices of the same size

    • (i,j)-th element:

  • Scalar multiplication:

    • results in the same size

  • Matrix subtraction:


Operations1

Operations

  • Multiplication of a row vector and a column vector

  • Matrix Multiplication

    • ,


Operations2

Operations


Operations3

Operations

  • Transpose:

    • (i,j)-the element:

    • transpose of a column vector: row vector

  • Rules:


Special matrices

Special Matrices

  • Square matrix:

  • Symmetric matrix:

  • Zero matrix

    • all elements are zeros

  • Identity Matrix:

    • each column (or row) standard basis

    • :


Operations4

Operations

  • (Square) Matrix Inverse

    • similar to inverse of a scalar:

    • inverse of a square matrix:

    • if there exists:

Non-singular


Operations5

Operations

  • Trace of a square matrix:

    • definition

    • rules


Today s topics2

Today’s Topics

  • Vectorand Matrix

  • Operation on Matrices/Vectors

  • Singular value decomposition

  • Norms

  • An Application in Text Analysis


Singular value decomposition

mm

mm

mn

mn

V is nn

V is nn

Singular Value Decomposition

  • A matrix:

  • Singular Value Decomposition (SVD)

  • The columns of are left singular vectors

  • The columns of are right singular vectors

  • is a diagonal matrix with singular values (positive values)


Singular value decomposition1

mm

mn

V is nn

Singular Value Decomposition

  • Illustration of SVD dimensions and sparseness


Singular value decomposition2

Singular Value Decomposition

  • Rank of a Matrix

    • organize singular values in descending order

    • the largest index that is non-zero


Eigen value decomposition

Eigen-value Decomposition

  • Eigenvectors(for a square mm matrix S)

  • Example

(right) eigenvector

eigenvalue


Eigen value decomposition1

Eigen-value Decomposition


Eigen value decomposition2

Eigen-value Decomposition

S = U *  * UT


Eigen value decomposition3

Eigen-value Decomposition

S = U *  * UT


Eigen value decomposition4

Eigen-value Decomposition

  • This is generally true for symmetric square matrix

  • Columns of U are eigenvectors of S

  • Diagonal elements of  are eigenvalues of S

S = U *  * UT


Eigen value decomposition5

mm

mn

V is nn

nn

nn

nn

Eigen-value Decomposition

  • A symmetric matrix:

  • Eigen-value Decomposition

  • The columns of are eigen-vectors

  • is a diagonal matrix with real eigen-values


Positive semi definite matrix

mm

mn

V is nn

nn

nn

nn

Positive (Semi-)Definite Matrix

  • A symmetric matrix:

  • Eigen-value Decomposition

  • The columns of are eigen-vectors

  • is a diagonal matrix with Positive eigen-values

  • is a diagonal matrix with Non-negative eigen-values


Today s topics3

Today’s Topics

  • Vectorand Matrix

  • Operation on Matrices/Vectors

  • Singular value decomposition

  • Norms

  • An Application in Text Analysis


Inner product

Inner Product

  • inner product between two vectors

  • Norm of a Vector: (Euclidean Norm, norm)


Inequalities

Inequalities

  • Cauchy-Schwarz Inequality

  • Triangle Inequality


P norm of a vector

p-Norm of a Vector

  • p-norm

    • p = 1 norm

    • p = 2 norm

    • p =  norm


Norm of a matrix

Norm of a Matrix

  • Inner Product between two matrices

  • Norm of a Matrix (Frobenius norm)


Other matrix norms

Other Matrix Norms

  • Induced Norm (operator norm):

    • p=2, spectral norm: maximum singular value

    • p=1, maximum absolute column sum

    • p= , maximum absolute row sum


Other matrix norms1

Other Matrix Norms

  • Schatten Norm:

    • p=1, trace norm (or nuclear norm)

    • p=2, Frobenius norm

    • p= , Spectral norm


Machine learning problems

Machine Learning Problems

  • Solve the following problems

Loss

norm


Today s topics4

Today’s Topics

  • Vectorand Matrix

  • Operation on Matrices/Vectors

  • Singular value decomposition

  • Norms

  • An Application in Search Engine


Search engine

Search Engine

  • A database of Webpages

  • A user-typed query

  • generate a list of relevant webpages

  • A ranking problem

https://www.facebook.com/

contain query words (LSI)

a lot of links to them (PageRank)


Representation of documents

Representation of documents

  • webpage is a document

  • document contains many terms (words)

  • To represent a document

    • collect all meaningful terms

    • count the occurrence of each term in a document


Representation of documents1

Representation of documents

  • Term-Document Matrix


Search engine1

Search Engine

  • Represent the query in the same way

    • e.g. query: “computer system”

Query

0

0

1

0

1

0

0

0

0

0

0

0


Search engine2

Search Engine

  • Retrieve Similar Documents

  • Query

  • Similarity

    • inner product

    • normalized inner product (cosine similarity)

    • Assume A is column normalized and q is normalized


Search engine3

Concept

Concept

Rep. of Concepts in term space

Rep. of concepts in document space

Search Engine

  • Latent Semantic Indexing

    • SVD


Search engine4

Search Engine

  • Low rank approximation:

    • approximate matrix with the largest singular values and singular vectors

Rank-k approximation


Search engine5

Search Engine

  • Why Low rank approximation:

    • data compression: billions to thousands

    • filter out noise

Rank-k approximation


Finding good concepts

Finding “Good Concepts”


Svd example m 2

X

X

SVD: Example: m=2


Lsi example m 2

X

X

LSI: Example: m=2


Lsi example m 21

X

X

LSI: Example: m=2


Lsi example m 22

X

X

LSI: Example: m=2


Lsi example m 3

LSI: Example: m=3

Top three left singular vectors

-0.2214 -0.1132 0.2890

-0.1976 -0.0721 0.1350

-0.2405 0.0432 -0.1644

-0.4036 0.0571 -0.3378

-0.6445 -0.1673 0.3611

-0.2650 0.1072 -0.4260

-0.2650 0.1072 -0.4260

-0.3008 -0.1413 0.3303

-0.2059 0.2736 -0.1776

-0.0127 0.4902 0.2311

-0.0361 0.6228 0.2231

-0.0318 0.4505 0.1411


Search engine6

Search Engine

  • Why Low rank approximation:

    • data compression: billions to thousands

    • filter out noise


  • Login