PRELIMINARIES

1. PRELIMINARIES CONTENTS • Linear Algebra • Convex Analysis Reference: Chapter 2 in BJS book.

2. Vectors Row or Column Vector: An n vector is a row or column array of n numbers. Each n vector can be represented by a point or by a line from the origin to the point, with an arrowhead at the end point of the line. Zero Vector: A vector whose every element is zero. ith Unit Vector: A vector whose every element is zero except the ith element which is 1. Sum Vector: The vector with all elements equal to 1. Denoted by “1”.

3. Vector Operations Addition of two vectors: a1 = (a11, a12, a13, …. , a1n) a2 = (a21, a22, a23, …. , a2n) a1 + a2 = (a11 + a21, a12 + a22, a13 + a23, …. , a1n+ a2n) Scalar multiplication: a = (a1, a2, a3, …. , an) ka = (ka1, ka2, ka3, …. , kan) Inner product (scalar product):   a = (a1, a2, a3, …. , an) b = (b1, b2, b3, …. , bn) ab = a1b1 + a2b2 + a3b3 + … + anbn

4. Vector Operations (contd.) Norm of a vector: Various norms (measures of size) of a vector can be used. lp norm of a vector a: Euclidean norm (l2 norm) : Euclidean space: An n-dimensional Euclidean space, denoted by En, is the collection of all vectors of dimension n

5. Linear and Affine Combinations A vector b in En is said to be a linear combination of a1, a2, …, ak in En, if (i) b = (ii) 1, 2, … , k are real numbers. A linear combination is said to be an affine combination if

6. Linear and Affine Subspaces A linear subspace SL of En is a subset of En such that if a1 and a2 SL, then every linear combination of a1 and a2 belongs to SL. An affine subspace SA of En is a subset of En such that if a1 and a2 SA, then every affine combination of a1 and a2 belongs to SA.

7. Linear Independence A collection of vectors a1, a2, … , ak of dimension n is called linearly independent if = 0 implies that lj = 0 for j = 1, 2, …, k A collection of vectors a1, a2, … , ak of dimension n is called linearly dependent if there exist lj for j = 1, 2, …, k, not all zero such that = 0

8. Spanning Set Spanning Set: A collection of vectors a1, a2, … , ak in En is said to span En if any vector in En can be represented as a linear combination of a1, a2, … , ak. In other words, given any vector b in En, we must be able to find scalars l1, l2, … , lk such that b = Sj=1,kljaj.

9. Basis Basis: A collection of vectors forms a basis of En if the following conditions hold: • a1, a2, … , ak span En. • If any of the vectors is removed, the remaining collection does not span En. Question: If we have a basis of En, what is the condition that will guarantee that if a vector of the basis, say aj, is replaced by another vector, say ap, then the new set of vectors still forms a basis?

10. Matrices A matrix is any rectangular array of numbers. The number in the ith row and jth column of a matrix A is called the ijth element of A and is denoted by aij. A typical mxn matrix: "mxn" is called the order of the matrix. Equality of Matrices: Two matrices A = [aij] and B = [bij] are said to be equal if and only if A and B are of the same order and aij = bij for all i and all j.

11. Matrix Operations Scalar Multiple of a Matrix: Addition of Two Matrices:

12. Transpose of a Matrix Transpose of a matrix A is denoted by AT.

13. Matrix Operations (contd.) The matrix product C = A B of an mxr matrix and rxn matrix B is the mxn matrix C defined as follows: ijth element of C = scalar product of row i of A and column j of B

14. Special Matrices Zero Matrix: An nx m matrix is called a zero matrix if each entry in the matrix is zero. Square Matrix : A matrix for which m = n. Identity Matrix In: An nxn square matrix for which aij = 1 if i = j, and aij = 0 if i≠j. Triangular Matrix: A square nxn matrix is called an upper triangular matrix if all the entries below the diagonal are zeros. Similarly, a square matrix is called a lower triangular matrix if all elements above the diagonal are zeros.

15. Systems of Linear Equations A linear system of m equations in n variables is: A solution to a system of m equations in n unknowns is a set of all possible values for the unknowns x that satisfies each of the system's m equations.

16. Matrix Representation of a System of Equations The above system of equations can be represented as Ax = b, where

17. Solving Systems of Linear Equations We will describe the Gauss-Jordan method for solving a system of linear equations. Using this method, we show that any system of linear equations must satisfy one of the following three cases: Case 1 : The system has no solution. Case 2 : The system has a unique solution. Case 3 : The system has an infinite number of solutions. The Gauss-Jordan method proceeds by performing three types of elementary row operations (ero).

18. Type 1 ero This ero transforms a given matrix A into a new matrix A' by multiplying a row of A by a nonzero scalar. Example: A'=3A:

19. Type 2 ero Multiply row i of A by a number c ≠ 0 and add to some row j ≠ i: row j of A' = c(row i of A) + (row j of A) Multiply row 2 of A by 4 and add to row 3.

20. Type 3 ero Interchange any two rows of A. For example, interchanging rows 1 and 3 of A, we obtain

21. Gauss – Jordan Method This method solves a system of linear equations by utilizing ero's in a systematic fashion. We illustrate the method using the following linear system: 2x1 + 2x2 + x3 = 9 2x1 - x2 + 2x3 = 6 x1 - x2 + 2x3 = 5 The augmented matrix representation of the above system:

22. Gauss – Jordan Method (contd.) By performing a sequence of ero's we can transform the above system to This system has a unique solution x1 = 1, x2 = 2 and x3 = 3. We transform the system column by column, starting at the first column and going up to the last column.

23. Gauss – Jordan Method (contd.)

24. When Do We Need Type 3 ero? When the diagonal element is zero.

25. System With No Solution x1 + 2x2 = 3 2x1 + 4x2 = 4 The above system has no solution. RESULT: If we apply the Gauss-Jordan method to a linear system and obtain a row of the form [0 0 ..... 0 | c] where c ≠ 0, then the original linear system has no solution.

26. System with Infinite Solutions x1 + x2 = 1 + x2 + x3 = 3 x1 + 2x2 + x3 = 4 x1 - x3 = -2 x2 + x3 = 3

27. Inverse of a Matrix For a given mxm matrix A, the mxm matrix B is the inverse of A if BA = AB = Im. The inverse matrix, if it exists, is unique. We denote the inverse of the matrix A by A-1. If A has an inverse, A is called nonsingular; otherwise it is called singular. Given an mxm matrix A, it has an inverse if and only if the rows of A are linearly independent or, equivalently, if the columns of A are linearly independent.

28. Calculation of the Inverse (contd.) The inverse matrix, if it exists, can be obtained through a finite number of ero’s. The elementary row operations that reduce A to the identity matrix, also reduce (A, I) to (I, A-1)

29. Example 1 (A-1 Exists) Consider the matrix A. Construct [A, I]

30. Example 1 (A-1 Exists) Divide the first row by 2. Add the new first row to the second row and subtract it from the third row.

31. Example 1 (A-1 Exists) (contd.) Multiply the second row by 2/5. Multiply the new second row to the second row by –1/2 and add to the first row. Multiply the new second row by 3/2 and add to the third row.

32. Example 1 (A-1 Exists) (contd.) Multiply the third row by 5/12. Multiply the new third row by –3/5 and add to the second row. Multiply the new third row by –1/5 and add to the first row.

33. Example 1 (A-1 Exists) Therefore, the inverse of A exists and is given by:

34. Example 1 (A-1 Does Not Exists) Consider the matrix A. Construct [A, I]

35. Example 1 (A-1 Does Not Exists) Multiply the first row by –2 and add to the second row. Multiply the first row by –1 and add to the third row.

36. Example 1 (A-1 Does Not Exists) Multiply the second row by –1/3. Then multiply the new second row by –1 and add to the first row. Multiply the new second row by –1 and add to the third row. There is no way that the left-hand side matrix can be transformed into the identity matrix by elementary row operations, hence the matrix A has no inverse.

37. Properties of the Inverse Matrix • If A is nonsingular, then At is also nonsingular and (At)-1 = (A-1)t. • If A and B are both n x n nonsingular matrices, then AB is • nonsingular and (AB)-1 = B-1A-1. • A matrix A is nonsingular if and only if its determinant (det(A)) is nonzero. • A triangular matrix (either lower or upper triangular) with nonzero diagonal elements has an inverse. This can be easily established by noting that such a matrix can be reduced to the identity by a finite number of elementary row operations. In particular, let D = diag {d1,…..,dn} be a diagonal matrix with diagonal elements d1,….,dn and all other elements being zero. If d1,….,dn are all nonzero,then D-1 = diag {1/d1,…..,1/dn}.

38. Rank of a Matrix Let A be an mxn matrix. The row rank of the matrix is equal to the maximum number of linearly independent rows of A. The column rank of A is the maximum number of linear independent columns of A. The row rank of A is always equal to its column rank.

39. Rank of a Matrix (contd.) How to determine the rank of a matrix?

40. Rank of a Matrix (contd.) rank (A)  minimum (n,m) If rank (A) = minimum (n,m), then A is said to be of full rank. The rank of A is k if and only if A can be reduced by performing ero’s to

41. Convex and Strict Convex Combinations For any two points x1 and x2 in X, any point x1 +(1- )x2 with 0    1, is called a convex combination of x1 and x2. A strict convex combination of x1 and x2 is any point x1 +(1- )x2 with 0 <  < 1.

42. Convex Functions A function f(x) is said to be a convex function if the following inequality holds for any two vectors x1 and x2: f(x1 + (1 - )x2)  f(x1)+ (1- )f(x2) for all 0    1 Loosely speaking, a function is a convex function if the line joining any two points is an overestimate of the function. Which of the following functions are convex functions? x2 x2 x2 x1 x1 x1

43. Concave Functions A function f(x) is said to be a concave function if the following inequality holds for any two vectors x1 and x2: f(x1 + (1 - )x2)  f(x1)+ (1- )f(x2) for all 0    1 Loosely speaking, a function is a concave function if the line joining any two points is an underestimate of the function. Which of the following functions are concave functions? x2 x2 x2 x1 x1 x1

44. Convex Sets • A set of points S is a convex set if the line segment joining any pair of points in S is wholly contained in S. • A set S is a convex set if every convex combination of any two points in the set is also in the set. That is, If x1Î S and x2Î S, then lx1 + (1-l)x2Î S for every 0 l 1

45. Extreme Points A point x in a convex set X is called an extreme point of X if and only if x cannot be represented as a strict convex combination of two distinct points in X. Give the extreme points of the following sets: