1 / 43

Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals

Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals . Syllabus.

osma
Download Presentation

Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Lecture 16 Nonlinear Problems:Simulated Annealing and Bootstrap Confidence Intervals

  2. Syllabus Lecture 01 Describing Inverse ProblemsLecture 02 Probability and Measurement Error, Part 1Lecture 03 Probability and Measurement Error, Part 2 Lecture 04 The L2 Norm and Simple Least SquaresLecture 05 A Priori Information and Weighted Least SquaredLecture 06 Resolution and Generalized Inverses Lecture 07 Backus-Gilbert Inverse and the Trade Off of Resolution and VarianceLecture 08 The Principle of Maximum LikelihoodLecture 09 Inexact TheoriesLecture 10 Nonuniqueness and Localized AveragesLecture 11 Vector Spaces and Singular Value Decomposition Lecture 12 Equality and Inequality ConstraintsLecture 13 L1 , L∞ Norm Problems and Linear ProgrammingLecture 14 Nonlinear Problems: Grid and Monte Carlo Searches Lecture 15 Nonlinear Problems: Newton’s Method Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals Lecture 17 Factor AnalysisLecture 18 Varimax Factors, Empircal Orthogonal FunctionsLecture 19 Backus-Gilbert Theory for Continuous Problems; Radon’s ProblemLecture 20 Linear Operators and Their AdjointsLecture 21 Fréchet DerivativesLecture 22 Exemplary Inverse Problems, incl. Filter DesignLecture 23 Exemplary Inverse Problems, incl. Earthquake LocationLecture 24 Exemplary Inverse Problems, incl. Vibrational Problems

  3. Purpose of the Lecture • Introduce Simulated Annealing • Introduce the Bootstrap Method • for computing Confidence Intervals

  4. Part 1Simulated Annealing

  5. Monte Carlo Methodcompletely undirectedNewton’s Methodcompletely directed

  6. Monte Carlo Methodcompletely undirectedNewton’s Methodcompletely directed slow, but foolproof fast, but can fall into local minimum

  7. compromise partially-directed random walk

  8. m(p) Elow Emedium Ehigh

  9. m(p) p(m*|m(p)) Elow Emedium Ehigh

  10. m* m(p) Elow Emedium Ehigh

  11. acceptance of m* as m(p+1) always accept in error is smalleraccept with probabilitywhere T is a parameterif error is bigger

  12. large T 1 always accept m* (undirected random walk) ignores the error completely

  13. small T 0 • accept m* only when error is smaller (directed random walk) strictly decreases the error

  14. intermediate T most iterations decrease the error but occasionally allow an m* that increases it

  15. large T undirected random walk m(p) Elow Emedium Ehigh

  16. small T directed random walk m(p) Elow Emedium Ehigh

  17. strategy start off with large T slowly decrease T during iterations undirected similar to Monte Carlo method (except more “local”) directed similar to Newton’s method (except precise gradient direction not used)

  18. strategy start off with large T slowly decrease T during iterations more random more directed claim is that this strategy helps achieve the global minimum

  19. analogous to annealing of metals high temperatures atoms randomly moving about due to thermal motions as temperature decreases atoms slowly find themselves in a minimum energy configuration orderly arrangement of a “crystal” www.sti-laser.com/technology/heat_treatments.html

  20. analogous to annealing of metals high temperatures atoms randomly moving about due to thermal motions as temperature decreases atoms slowly find themselves in a minimum energy configuration orderly arrangement of a “crystal” hence “simulated annealing” and T called “temperature”

  21. this is just Metroplois-Hastings (way of producing realizations of a random variable) applied to the p.d.f.

  22. this is just Metroplois-Hastings (way of producing realizations of a random variable) applied to the p.d.f. sampling a distribution that starts out wide and blurry but sharpens up as T is decreases

  23. (A) (B) (C)

  24. for k = [1:Niter] T = 0.1 * Eg0 * ((Niter-k+1)/Niter)^2; ma(1) = random('Normal',mg(1),Dm); ma(2) = random('Normal',mg(2),Dm); da = sin(w0*ma(1)*x) + ma(1)*ma(2); Ea = (dobs-da)'*(dobs-da); if( Ea < Eg ) mg=ma; Eg=Ea; p1his(k+1)=1; else p1 = exp( -(Ea-Eg)/T ); p2 = random('unif',0,1); if( p1 > p2 ) mg=ma; Eg=Ea; end end end

  25. Part 2Bootstrap Method

  26. theory of confidence intervals error is the data result in errors in the estimated model parameters m(d) p(m) p(d) d m dobs mest

  27. theory of confidence intervals error is the data result in errors in the estimated model parameters m(d) p(m) p(d) 2½% 2½% 2½% 2½% d m 95% confidence 95% confidence

  28. Gaussian linear theoryd = Gmm = G-gdstandard error propagation[covm]=G-g [covd] G-gTunivariate Gaussian distribution has95% of error within two σ of its mean

  29. What to do withGaussian nonlinear theory?One possibilitylinearize theory and use standard error propagationd = g(m)m-m(p) ≈G(p) –g [d- g(m(p)) ][covm]≈G(p)-g [covd] G(p)-g

  30. disadvantagesunknown accuracyandneed to compute gradient of theory G(p)G(p) not computed when using some solution methods

  31. alternativeconfidence intervals withrepeat datasets do the whole experiment many times use results of each experiment to make compute mest • create histograms from many mest’s • derive empirical 95% confidence intervals • from histograms

  32. Bootstrap Method create approximate repeat datasets by randomly resampling (with duplications) the one existing data set

  33. example of resampling original data set random integers in range 1-6 resampleddata set

  34. example of resampling original data set random integers in range 1-6 new data set

  35. example of resampling original data set random integers in range 1-6 resampleddata set note repeats

  36. rowindex = unidrnd(N,N,1); xresampled = x( rowindex ); dresampled = dobs( rowindex );

  37. interpretation of resampling mixing sampling duplication p(d) p’(d)

  38. (A) (B) p(m1) m1 (C) m1 p(m2) m2 m2

  39. Nbins=50; m1hmin=min(m1save); m1hmax=max(m1save); Dm1bins = (m1hmax-m1hmin)/(Nbins-1); m1bins=m1hmin+Dm1bins*[0:Nbins-1]'; m1hist = hist(m1save,m1bins); pm1 = m1hist/(Dm1bins*sum(m1hist)); Pm1 = Dm1bins*cumsum(pm1); m1low=m1bins(find(Pm1>0.025,1)); m1high=m1bins(find(Pm1>0.975,1));

  40. Nbins=50; m1hmin=min(m1save); m1hmax=max(m1save); Dm1bins = (m1hmax-m1hmin)/(Nbins-1); m1bins=m1hmin+Dm1bins*[0:Nbins-1]'; m1hist = hist(m1save,m1bins); pm1 = m1hist/(Dm1bins*sum(m1hist)); Pm1 = Dm1bins*cumsum(pm1); m1low=m1bins(find(Pm1>0.025,1)); m1high=m1bins(find(Pm1>0.975,1)); histogram

  41. Nbins=50; m1hmin=min(m1save); m1hmax=max(m1save); Dm1bins = (m1hmax-m1hmin)/(Nbins-1); m1bins=m1hmin+Dm1bins*[0:Nbins-1]'; m1hist = hist(m1save,m1bins); pm1 = m1hist/(Dm1bins*sum(m1hist)); Pm1 = Dm1bins*cumsum(pm1); m1low=m1bins(find(Pm1>0.025,1)); m1high=m1bins(find(Pm1>0.975,1)); empirical p.d.f.

  42. Nbins=50; m1hmin=min(m1save); m1hmax=max(m1save); Dm1bins = (m1hmax-m1hmin)/(Nbins-1); m1bins=m1hmin+Dm1bins*[0:Nbins-1]'; m1hist = hist(m1save,m1bins); pm1 = m1hist/(Dm1bins*sum(m1hist)); Pm1 = Dm1bins*cumsum(pm1); m1low=m1bins(find(Pm1>0.025,1)); m1high=m1bins(find(Pm1>0.975,1)); empirical c.d.f.

  43. Nbins=50; m1hmin=min(m1save); m1hmax=max(m1save); Dm1bins = (m1hmax-m1hmin)/(Nbins-1); m1bins=m1hmin+Dm1bins*[0:Nbins-1]'; m1hist = hist(m1save,m1bins); pm1 = m1hist/(Dm1bins*sum(m1hist)); Pm1 = Dm1bins*cumsum(pm1); m1low=m1bins(find(Pm1>0.025,1)); m1high=m1bins(find(Pm1>0.975,1)); 95% confidence bounds

More Related