Direct Methods for Linear Systems

1. Direct Methods for Linear Systems Lecture 3 Alessandra Nardi Thanks to Prof. Jacob White, Suvranu De, Deepak Ramaswamy, Michal Rewienski, and Karen Veroy

2. Last lecture review • Formulation of circuit equations • Conservation laws (KCL, KVL) • Branch constitutive equations (BCE) • KCL, KVL, BCE combined in different ways: • STA • MNA

3. Outline • Systems of linear equations • Existence and uniqueness review • Gaussian Elimination basics • LU factorization • Pivoting

4. Systems of linear equations • Problem to solve: M x = b • Given M x = b : • Is there a solution? • Is the solution unique?

5. Systems of linear equations Find a set of weights x so that the weighted sum of the columns of the matrix M is equal to the right hand side b

6. Systems of linear equations - Existence A solution exists when b is in the span of the columns of M A solution exists if: There exist weights, x1, …., xN, such that:

7. Systems of linear equations - Uniqueness Suppose there exist weights, y1, …., yN, not all zero, such that: Then: Mx = b  Mx + My= b  M(x+y) = b A solution is unique only if the columns of M are linearlyindependent.

8. Systems of linear equations Square matrices • Given Mx = b, where M is square • If a solution exists for any b, then the solution for a specific b is unique. For a solution to exist for any b, the columns of M must span all N-length vectors. Since there are only N columns of the matrix M to span this space, these vectors must be linearly independent. A square matrix with linearly independent columns is said to be nonsingular.

9. Application Problems • Matrix is n x n • Often symmetric and diagonally dominant • Nonsingular of real numbers

10. Methods for solving linear equations • Direct methods: find the exact solution in a finite number of steps • Iterative methods: produce a sequence a sequence of approximate solutions hopefully converging to the exact solution

11. Gaussian Elimination Basics Gaussian Elimination Method for Solving M x = b • A “Direct” Method Finite Termination for exact result (ignoring roundoff) • Produces accurate results for a broad range of matrices • Computationally Expensive

12. Gaussian Elimination Basics Reminder by 3x3 example

13. Gaussian Elimination Basics – Key idea Use Eqn 1 to Eliminate x1 from Eqn 2 and 3

14. Pivot MULTIPLIERS GE Basics – Key idea in the matrix Remove x1 from eqn 2 and eqn 3

15. Pivot Multiplier GE Basics – Key idea in the matrix Remove x2 from eqn 3

16. GE Basics – Simplify the notation Remove x1 from eqn 2 and eqn 3

17. Pivot Multiplier GE Basics – Simplify the notation Remove x2 from eqn 3

18. Altered During GE GE Basics – GE yields triangular system ~ ~

19. GE Basics – Backward substitution

20. GE Basics – RHS updates

21. GE basics: summary (1)M x = b U x = y Equivalent system U: upper trg (2) Noticed that: Ly = b L: unit lower trg • U x = y LU x = b  M x = b GE  Efficient way of implementing GE: LU factorization

22. M = L U = Gaussian Elimination Basics Solve M x = b Step 1 Step 2 Forward Elimination Solve L y = b Step 3 Backward Substitution Solve U x = y Note: Changing RHS does not imply to recompute LU factorization

23. GE Basics – Fitting the pieces together

24. GE Basics – Fitting the pieces together

25. LU factorization Basics – Picture

26. Pivot Multiplier LU BasicsSource-row oriented approach algorithm For i = 1 to n-1 {“For each source row” For j = i+1 to n {“For each target row below the source” For k = i+1 to n {“For each row element beyond Pivot” } } }

27. LU BasicsTarget-row oriented approach algorithm For i = 2 to n {“For each target row” For j = 1 to i-1 {“For each source row above the target” For k = j+1 to n {“For each row element beyond Pivot” } } } Pivot Multiplier

28. k k Factored Portion Multipliers Factored Portion Mult k k Active Set Active Set LU – Source-row and Target-row Source-Row oriented approach Target-Row oriented approach

29. Pivot multipliers Multiplier Multiply-adds LU Basics – Computational Complexity For i = 1 to n-1 {“For each Row” For j = i+1 to n {“For each target Row below the source” For k = i+1 to n {“For each Row element beyond Pivot” } } }

30. LU Basics – Limitations of the naïve approach • Zero Pivots • Small Pivots (Round-off error) • both can be solved with partial pivoting

31. LU Basics – Partial pivoting for zero pivots At Step i Multipliers Factored Portion Row i (L) Row j What if Cannot form Simple Fix (Partial Pivoting) If Find Swap Rowj with i

32. LU Basics – Partial pivoting for zero pivots Two Important Theorems • ) Partial pivoting (swapping rows) always succeeds if M is non singular • ) LU factorization applied to a diagonally dominant matrix will never produce a zero pivot

33. LU Basics – Partial pivoting for small pivots GE

34. LU Basics – Partial pivoting for small pivots GE Rounded to 3 digits

35. LU Basics – Partial pivoting for small pivots An Aside on Floating Point Arithmetic Double precision number 64 bits sign 52 bits 11 bits Basic Problem • Avoid sum and subtraction of large and tiny numbers •  Avoid big multipliers

36. LU Basics – Partial pivoting for small pivots Partial Pivoting for Roundoff reduction Small Multipliers

37. LU Basics – Partial pivoting for small pivots swap GE Rounded to 3 digits

38. k k k k Pivoting strategies • Partial Pivoting: • Only row interchange • Complete Pivoting • Row and Column interchange • Threshold Pivoting • Only if prospective pivot is found to be smaller than a certain threshold

39. Summary • Existence and uniqueness review • Gaussian elimination basics • GE basics • LU factorization • Pivoting