1 / 49

Orthogonalization via Deflation By Achiya Dax Hydrological Service Jerusalem , Israel

Orthogonalization via Deflation By Achiya Dax Hydrological Service Jerusalem , Israel e-mail: dax20@water.gov.il. Overview * Motivation: The Symmetric Case * Rectangular Quotients * Retrieval of Singular Vectors * Rectangular Iterations

malaya
Download Presentation

Orthogonalization via Deflation By Achiya Dax Hydrological Service Jerusalem , Israel

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Orthogonalization via Deflation By Achiya Dax Hydrological Service Jerusalem , Israel e-mail: dax20@water.gov.il

  2. Overview *Motivation: The Symmetric Case * Rectangular Quotients * Retrieval of Singular Vectors * Rectangular Iterations * Orthogonalization via Deflation * Applications

  3. The Symmetric Case S = ( sij ) a symmetric positive semi-definite n x n matrix With eigenvalues l1 ³ l2 ³ ... ³ ln ³ 0 and eigenvectors v1 , v2 , … , vn S vj =ljvj, j = 1, … , n . S V = V D V = [v1 , v2 , … , vn] , VTV = V VT = I D = diag {l1,l2,…,ln} S = V D VT = SljvjvjT

  4. Low-Rank Approximations S =l1v1v1T+ … +lnvnvnT T1=l1v1v1T T2=l1v1v1T+l2v2v2T . . . Tk=l1v1v1T+l2v2v2T+ … +lkvkvkT Tkis a low-rank approximation of order k .

  5. The Euclidean vector norm u = (u1 , u2 , … , un)T ||u||2= ( uTu ) ½ = ] S uj 2½[ The Frobenius matrix norm A = ( aij ) , ||A|| F] = S S| aij|2½[

  6. AMinimumNormProblem Let the vector v* solve the minimum norm problem minimize E(v) =||S- vvT||F 2 . Then v1 = v* / || v* ||2andl1 = (v*)Tv* constitutes a dominant eigenpair of S .

  7. The Rayleigh Quotient r=r (v,S) =vTSv/vTv solves the minimum norm problem minimize f(q) =||Sv - qv||2 r estimates an eigenvalue corresponding to V

  8. ASimple Error Bound Given r=r (v,S) =vTSv/vTv there exists an eigenvalue l of S such that |l - r| £ ||Sv - rv||2

  9. A Second Minimum Norm Problem Given any unit vector u , ||u||2=1 , the Rayleigh Quotient r(u) =uTSu/uTu=uTSu solves the one parameter problem minimizef (q) =|| S- quuT || F2

  10. The Power Method Starting with some unit vector p0 . The kth iteration, k = 1, 2, 3, … , Step 1:Computewk=Spk-1 Step 2:Computerk= (pk-1)Twk Step 3: Normalizepk = wk/||wk||2

  11. THE POWER METHOD Asymptotic Rates of Convergence ( Assuming l1>l2 ) {pk}av1at a linear rate, proportionaltol2/l1 {rk}al1at a linear rate, proportionalto (l2/l1)2 Monotony:l1³…³rk³… ³r2³r1> 0

  12. THE POWER METHOD The asymptotic rates of convergence depend on the ratio l2/l1 and can be arbitrarily slow. Yet rk provides a fair estimate of l1 within a few iterations ! For a “worst case analysis” see D.P. O’Leary, G.W. Stewart and J.S. Vandergraft, “Estimating the largest eigenvalue of a positive definite matrix”, Math. of Comp., 33(1979), pp. 1289 – 1292.

  13. THE POWER METHOD An eigenvector vj is called “large” if lj³l1/2and “small” if lj<l1/2 . Inmost of the practical situations, for “small” eigenvectors pkTvj becomes negligible after a small number of iterations. Thus, after a few iterationspkactually lies in a subspace spanned by “large” eigenvectors.

  14. Deflation by Subtraction S =l1v1v1T+ … +lnvnvnT. S1= S -l1v1v1T= l2 v2v2T+ … + lnvnvnT . S2= S1-l2v2v2T= l3v3v3T+ … +lnvnvnT . . . . Sn-1=lnvnvnT . Sn= 0 . Hotelling (1933, 1943)

  15. Can we extend these tools to rectangular matrices?

  16. The Rectangular Case A = (aij) a real m x n matrix , p = min{m,n} With singular values s1³s2³…³sp³ 0 , Left singular vectors u1 , u2 , … , up Right singular vectors v1 , v2 , … , vp Avj=sjuj , ATuj=sjvj = 1,…,p .

  17. The Singular Value Decomposition A = U S VT S= diag {s1,s2,…,sp} , p = min{m,n} U = [u1 , u2 , … , up] , UTU = I V = [v1 , v2 , … , vp] , VTV = I AV = US ATU = VS Avj=sjuj , ATuj=sjvjj=1, … , p .

  18. Low - Rank Approximations A = U S VT=SsjujvjT A=s1u1v1T+ s2u2v2T+ … +spupvpT . B1=s1u1v1T B2=s1u1v1T+ s2u2v2T . . . Bk=s1u1v1T+ s2u2v2T+ … +skukvkT Bk is a low-rank approximation of order k . (Also called "truncated SVD“ or “filtered SVD”.)

  19. A Minimum Norm Problem Let the vectors u* and v* solve the problem minimize F(u,v) =||A- uvT||F2 then u1 = u* / || u* ||2 , v1 = v* / || v* ||2 , and s1 = || u* ||2 || v* ||2 ( See the Eckhart-Young Theorem.)

  20. The Rectangular Quotient Given two vectors , u and v , the Rectangular Rayleigh Quotient r(u,v) =uTAv/||u||2||v||2 estimates the “corresponding” singular value.

  21. The Rectangular Rayleigh Quotient Given two unit vectors , u and v , the Rectangular Rayleigh Quotient r(u,v) =uTAv/||u||2||v||2 solves the following three problems minimize f1(q) =||A- quvT||F minimize f2(q) =||Av - qu||2 minimize f3(q) =||ATu - qv||2

  22. Approximating a left singular vector Given a right singular vector, v1, the corresponding left singular vector, u1, is attained by solving the least norm problem minimize g(u)=||A- uv1T||F2 That is, u1 = Av1/v1Tv1. (The rows of A are orthogonalized against v1T .)

  23. Approximating a right singular vector Given a left singular vector , u1 , the corresponding right singular vector , v1 , is attained by solving the least norm problem minimize h(v)=||A– u1vT||F2 That is, v1 = ATu1/u1Tu1. (The columns of A are orthogonalized against u1.)

  24. Rectangular Iterations - Motivation The kth iteration , k = 1, 2, 3, … , starts with uk-1 and vk-1 and ends with uk and vk . Given vk-1 the vector uk is obtained by solving the problem minimize g(u) =||A-uvk-1T||F2. That is, uk = Avk-1 / vk-1Tvk-1 . Then, vk is obtained by solving the problem minimize h(v) =||A- ukvT||F2 , which gives vk=ATuk/ ukTuk.

  25. Rectangular Iterations – Implementation The kth iteration , k = 1, 2, 3, … , uk=Avk-1/vk-1Tvk-1, vk=ATuk/ukTuk . The sequence {vk/||vk||2} is obtained by applying the Power Method on the matrix ATA . The sequence{uk/||uk||2} is obtained by applying the Power Method on the matrix AAT .

  26. Left Iterations uk=Avk-1/vk-1Tvk-1, vk=ATuk/ukTuk. ------------------------------------------------------------------------------------------------------- vkTvk=vkTATuk/ukTuk Right Iterations vk=ATuk-1/uk-1Tuk-1 , uk=Avk/vkTvk. ------------------------------------------------------------------------------------------------------ ukTuk=ukTAvk/vkTvk Can one see a difference?

  27. Some Useful Relations In both cases we have ukTuk vkTvk=ukTAvk , ||uk||2||vk||2=ukTAvk/||uk||2||vk||2=r(uk,vk) , and h(uk,vk)=ukTAvk/ukTukvkTvk=1 . The objective functionF(u,v) =||A -uvT||F2 satisfiesF(uk,vk) =||A||F2-ukTukvkTvk and F(uk,vk) -F(uk+1 ,vk+1) = = uk+1Tuk+1vk+1Tvk+1 -ukTukvkTvk > 0

  28. Convergence Properties Inherited from the Power Method, assuming s1>s2. The sequences {uk/||uk||2} and {vk/||vk||2} converge at a linear rate, proportional to (s2/s1)2 . {ukTukvkTvk} a (s1)2 at a linear rate, proportional to (s2/s1)4 Monotony: (s1)2³uk+1Tuk+1vk+1Tvk+1³ ukTukvkTvk > 0

  29. Convergence Properties rk = ||uk||2||vk||2 provides a fair estimate ofs1 within a few rectangular iterations !

  30. Convergence Properties After a few rectangular iterations {rk , uk ,vk} provides a fair estimate of a dominant triplet {r1 , u1 ,v1} .

  31. Deflation by Subtraction A1= A =s1u1v1T+ … +spupvpT . A2= A1-s1u1v1T=s2u2v2T+ … +spupvpT A3= A2-s2u2v2T=s3u3v3T+ … +spvpvpT . . . Ak+1= Ak-skukvkT=sk+1uk+1vk+1T+…+spupvpT . . .

  32. Deflation by Subtraction A1= A A2= A1-s1u1v1T A3= A2-s2u2v2T . . . Ak+1= Ak-skukvkT . . . where { sk , uk ,vk } denotes a computed dominant singular triplet of Ak .

  33. The Main Motivation At the kth stage , k = 1, 2, … , a few rectangular iterations provide a fair estimate of adominant tripletof AK .

  34. Low - Rank Approximation Via Deflation s1³s2³ … ³sp³ 0, A=s1u1v1T+ s2u2v2T+ … +spupvpT . B1=s*1u*1v*1T ( * means computed values ) B2=s*1u*1v*1T+ s*2u*2v*2T . . . Bl= s*1u*1v*1T+ s*2u*2v*2T+ …+s*lu*lv*lT Bl is a low - rank approximation of order l . ( Also called "truncated SVD“ orthe “filtered part” of A . )

  35. Low - Rank Approximation of Order l A=s1u1v1T+ s2u2v2T+ … +spupvpT . Bl= s*1u*1v*1T+ s*2u*2v*2T+ …+s*lu*lv*lT Bl= UlSlVlT Ul= [u*1,u*2,…,u*l] , Vl= [v*1,v*2,…, v*l] , Sl= diag{s*1 , s*2 , … , s*l} ( * means computed values )

  36. What About Orthogonality ? Does UlTUl= I and VlTVl= I? The theory behind the Power Method suggests that the more accurate are the computed singular triplets the smaller is the deviation from orthogonality . Is there a difference ( regarding deviation from orthogonality ) between Ul and Vl?

  37. Orthogonality Properties ( Assuming exact arithmetic . ) Theorem 1 : Consider the case when each singular triplet, {s*j,u*j,v*j} , is computed by a finite number of "Left Iterations". (At least one iteration for each triplet. ) In this case UlTUl = IandUlTAl = 0 regardless the actual number of iterations !

  38. Left Iterations uk=Avk-1/vk-1Tvk-1, vk=ATuk/ukTuk. Right Iterations vk=ATuk-1/uk-1Tuk-1 , uk=Avk/vkTvk. Can one see a difference?

  39. Orthogonality Properties ( Assuming exact arithmetic . ) Theorem 2 : Consider the case when each singular triplet, {s*j,u*j,v*j} , is computed by a finite number of “Right Iterations". (At least one iteration for each triplet. ) In this case VlTVl = I and AlVl = 0 regardless the actual number of iterations !

  40. Finite Termination Assuming exact arithmetic, r=rank(A) . Corollary :In both caseswe have A = Br = s*1u*1v*1T + … + s*ru*rv*rT, regardless the number of iterations per singular triplet !

  41. A New QR Decomposion Assuming exact arithmetic ,r=rank(A) . In both caseswe obtain an effective “rank–revealing”QR decomposition A = UrSr VrT. In “Left Iterations”UrTUr = I . In “Right Iterations” VrTVr = I.

  42. The Orthogonal Basis Problem Is to compute an orthogonal basisof Range(A). The Householder and Gram-Schmidt orthogonalizations methods use a “column pivoting for size” policy, which completely determine the basis.

  43. The Orthogonal Basis Problem The new method , “Orthogonalization via Deflation” , has larger freedom in choosing the basis. At the kth stage, the ultimate choice for a new vector to enterthe basis is uk , the kth left singular vector of A . ( But accurate computation of uk can be “too expensive”. )

  44. The Main Theme At the kth stage , a few rectangular iterations are sufficient to provide a fair subtitute of uk .

  45. Applications *Missing data reconstruction. * Low-rank approximations of large sparse matrices. * Low-rank approximations of tensors.

  46. Applications in Missing Data Reconstruction Consider the case when some entries of A are missing. * Missing Data in DNA Microarrays * Tables of Annual Rain Data * Tables of Water Levels in Observation Wells * Web Search Engines Standard SVD algorithms are unable to handle such matrices. The Minimum Norm Approach is easily adapted to handle matrices with missing entries.

  47. A Modified Algorithm The objective function F(u,v) =||A- uvT||F2 is redefined as F(u,v) =SS( aij – uivj )2 , where the sum is restricted to known entries of A . ( As before, u= (u1, u2, … , um)Tandv= (v1, v2, … , vn)T denote the vectors of unknowns. )

  48. Overview *Motivation: The Symmetric Case * Rectangular Quotients * Retrieval of Singular Vectors * Rectangular Iterations * Orthogonalization via Deflation * Applications

  49. The END Thank You

More Related