1 / 28

Multivariate Statistics

Multivariate Statistics. Matrix Algebra II W. M. van der Veld University of Amsterdam. Overview. The determinant of a matrix The matrix inverse System of equations. The determinant of a matrix.

nura
Download Presentation

Multivariate Statistics

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Multivariate Statistics Matrix Algebra II W. M. van der Veld University of Amsterdam

  2. Overview • The determinant of a matrix • The matrix inverse • System of equations

  3. The determinant of a matrix • The determinant of a matrix is a scalar and is denoted as |A| or det(A). Det(A) only exists when A is a square matrix. • It has very important mathematical properties, but it is very difficult to provide a substantive definition. • The determinant is necessary to compute the inverse of a matrix (A-1). • The inverse of a matrix is needed for solving systems of linear equations; multivariate statistics often comes down to this. • When the determinant is zero, there exists no solution to a system of linear equations. • Let’s see how the value of the determinant is found.

  4. Cofactors The determinant of a matrix • How to do it? The most simple case, a 2 by 2 matrix . • Det(A)=|A|=?

  5. The determinant of a matrix • One step further, a 3 by 3 matrix. • Det(A)=|A|=? Cofactor

  6. The determinant of a matrix • You should have noted that for matrices larger than first order, computation of the determinant is a recursive process. This process stops each time a 1 by 1 determinant is encountered, and involves multiplication by the cofactors.

  7. The determinant of a matrix • Let A be a matrix of order n x n. If we omit one or more rows or columns from A, we obtain a matrix of smaller order, called a minor of the matrix. • Similarly, we have minors of a determinant, and in particular, if we omit from the determinant the ith row and the jth column, the resulting minor will be square and its determinant will be symbolized |Mij|. This determinant is called a cofactor (cij) if we give it a sign equal to (-1)i+j, so that: cij = (-1)i+j |Mij|. Using this notation we can write a formula for the expansion of a determinant of order n: In this version the determinant is expanded according to it’s ith row.

  8. The determinant of a matrix • The following rules are important for determinants, and can help you sometimes to simplify calculations: • The determinant of A has the same value as the determinant of A’. • The value of the determinant changes sign if one row (column) is interchanged with another row (column). • If a determinant has two equal rows (columns), its value is zero. • If a determinant has two rows (columns) with proportional elements, its value is zero. • If all elements in a row (column) are multiplied by a constant, the value of the determinant is multiplied by that constant. • If a determinant has a row (column) in which all elements are zero, the value of the determinant is zero. • The value of the determinant remains unchanged if one row (column) is added to or subtracted form another row (column). Moreover, if a row (column) is multiplied by a constant and then added to or subtracted from another row (column) the value remains unchanged.

  9. The determinant of a matrix • What is the determinant of:

  10. The matrix inverse • Let A be a square matrix. If we can find a matrix B of the same order as A such that AB=BA=I, then B is said to be the inverse of A and is symbolized A-1. A-1, if exists, can be found as follows. • Let C be the matrix of cofactors of A (i.e., cij is the cofactor obtained from the minor |Mij|); then • Where C’ is the transpose of C (or if one prefers, C’ is the matrix of cofactors of A’). It is immediately seen that the inverse is undefined if A is not square (since then there is no determinant |A|), and also if |A| is equal to zero.

  11. The matrix inverse • Illustration that AA-1 = A-1A = I.

  12. Compute determinant The matrix inverse • How did I get A-1? Now Compute C C transpose => C’ Calculate A-1

  13. The matrix inverse • Another way to calculate A-1. This way introduces you to solving systems of equations.

  14. The matrix inverse • Rules for algebra with inverse matrices: • AA-1 = A-1A = I • (AB)-1 = B-1A-1 • (ABC)-1 = C-1B-1A-1 • Proof that(AB)-1 = B-1A-1.

  15. System of equations • In the introduction I already mentioned that the basic linear equation y=bx will be very important for multivariate methods. • Here we will discuss how to solve systems of such linear equations.

  16. System of equations • Illustration. Suppose we have the following set of equations:-3=1x1+4x2 1=3x1+2x2 • The basic way to think about this problem set is finding the intersection, i.e. for which unknowns are the equations satisfied. • This can be solved in a simple way (old style). • The solution is basically the intersection of the lines represented by the equation. • You won’t be surprised that there is a more general way to solve systems of linear equations, using matrix algebra.

  17. System of equations • Solution for m equations with n unknowns: m=n. • What to do? Normally you divide by A so that you obtain a solution for x (give example: 15=3x). • Matrix division is defined as multiplication by the inverse, so:

  18. System of equations • Example. Suppose we have the following set of equations:-3=1x1+4x21=3x1+2x2 • We already solved this one, resulting in x1=1 and x2=-1. • The set of equations can be written as a matrix operation.

  19. System of equations • Thus, we have to find the inverse of: A => A-1 = C’/|A| • We have to take the transpose of C

  20. System of equations • We have to divide by |A|. • Thus the inverse matrix is.

  21. System of equations • Thus a solution for: -3=1x1+4x2 1=3x1+2x2 is found via

  22. System of equations Exercise, solve: x1 + 2x2 = 0 3x1 + 7x2 = 1 • So if Ax = k solve via x = A-1k. • .... But it is not always so simple … A-1Ax = Ix = x = A-1k

  23. System of equations • Sometimes, the requirement that m=n seems to be fulfilled, so that there should exist a solution. • But consider the following cases. (Row 2 = 2 x Row 1) (Row 3 = Row 1 + Row 2) (Column 3 = Column 1 + Column 2), etc.

  24. System of equations • These situations are called linear dependence: • Given vectors: x1, x2,…, xn-1 • Another vector xn is linearly dependent if there exists constants α1, α2,…, αn-1 such that:xn= α1x1+α2x2+ …+αn-1xn-1 • Otherwise the vector xn is linearly independent. • In case of linear dependence; |A|= 0. • And then the inverse is not defined: A-1=C’/|A|. • And when the inverse is not defined we cannot find a solution via: A-1k=x.

  25. System of equations • Generally a unique solution exists only if m=n, and |A|≠0 • When are there ‘problems’? • If m<n there are many solutions, the problem is underdetermined. 8x1+10x2+14x3=94x1+12x2+16x3=10 • if m>n there are no solutions, the problem is overdetermined.8x1+10x2=94x1+12x2=104x1+10x2=2

  26. System of equations • Using the idea of linear dependency, the rank of a matrix can be introduced. • rank(A) = number of linearly independent rows or columns. • Given an mxn matrix, with m ≥ n, then if • |A| ≠ 0  rank(A) = n full rank, solvable • |A| = 0  rank(A) < n rank deficient • We will get back to the issue of rank.

  27. Overdetermined Systems • Find Ax “closest” to k • Least-squares distance measure • Minimization problem: • Normal equations: (A’A)x = A’k • Solution: x = (A’A)-1A’k • A’A must be nonsingular; i.e. |A’A|≠0 • (A’A)-1A’ is called the left inverse matrix

  28. Constrained minimization problem: Underdetermined Systems • Find “smallest” x that satisfies equations • Minimum norm objective • Solution: x = A’(AA’)-1k • AA’ must be nonsingular • A’(AA’)-1 is called the right inverse

More Related