html5-img
1 / 22

Confidence Interval Estimation in System Dynamics Models

Confidence Interval Estimation in System Dynamics Models. Gokhan Dogan* MIT Sloan School of Management System Dynamics Group. *Special thanks to John Sterman for his support. Motivation. Calibration. Manual Calibration. Automated Calibration (e.g. Vensim, Powersim).

delta
Download Presentation

Confidence Interval Estimation in System Dynamics Models

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Confidence Interval Estimation in System Dynamics Models Gokhan Dogan* MIT Sloan School of Management System Dynamics Group *Special thanks to John Sterman for his support

  2. Motivation Calibration Manual Calibration Automated Calibration (e.g. Vensim, Powersim)

  3. Motivation • Once model parameters are estimated with automated calibration, next step: Estimate confidence intervals! • Questions: -Are there available tools at software packages? -Do these methods have any limitations? -Are there alternative methods?

  4. Why are confidence intervals important We reject the claim that the parameter value is equal to 0 (with 95% probability) We can’t reject the claim that the parameter value is equal to 0 (with 95% probability) 95% Confidence Interval 95% Confidence Interval 0 0 Parameter Estimate θ Parameter Estimate θ

  5. How can we estimate confidence intervals? Used in the System Dynamics Software (Vensim) /Literature The method we suggest for System Dynamics models Likelihood Ratio Method Bootstrapping Both methods yield approximate confidence intervals!

  6. Likelihood Ratio Method • The likelihood ratio method is used in system dynamics software packages (Vensim) and literature (Oliva and Sterman, 2001). • It relies on asymptotic theory (large sample assumption).

  7. However

  8. Bootstrapping • Introduced by Efron (1979) and based on resampling. Extensive survey in Li and Maddala (1996). • It seems more appropriate for system dynamics models because - It doesn’t require large sample - It is applicable when there is feedback (autocorrelation) - It doesn’t assume normally distributed error terms

  9. Drawbacks of bootstrapping • The software packages do not implement it. • It is time consuming.

  10. Bootstrapping Fit the model and estimate parameters Compute the Error Terms

  11. Bootstrapping uses resampling Nonparametric: Reshuffle Them and Generate many many new error term sets using the autocorrelation information Parametric: Fit a distribution and Generate many many new error term sets using the autocorrelation and distribution information

  12. Resampling the Error Terms • If we know that: - The error terms are autocorrelated - Their variance is not constant (heteroskedasticity) - They are not normally distributed => We can use this information while resampling the error terms • Flexibility of bootstrapping stems from this stage

  13. . . . . . . FABRICATED ERROR TERMS FABRICATED “HISTORICAL” DATA + =

  14. . . . FABRICATED “HISTORICAL” DATA Fit the model and estimate parameters Parameter Estimate Parameter Estimate Fit the model and estimate parameters Parameter Estimate 500 Parameter Estimates Fit the model and estimate parameters

  15. Distribution of a model parameter

  16. Experiments • We had experimental time series data from 240 subjects. • Subjects were beer game players. • For each subject we had 48 data points, so we estimated parameters and confidence intervals using 48 data points.

  17. Model (Same as Sterman 1989) • Ot = Max[0, θLRt + (1–θ)ELt + α(S' – St –βSLt) + error termt] Parameters to be estimated are θ,α, β, S‘

  18. Individual Results 95% Confidence Intervals for θ Likelihood Ratio Method 0 95% CI 1 0.77 θ=0.95 Bootstrapping 95% CI 0 1 0.01 θ=0.95

  19. Individual Results 95% Confidence Intervals for β Likelihood Ratio Method 0 95% CI 0.2 Significantly Different From 0!!! β =0.01 Bootstrapping 95% CI 0 0.2 β =0.01

  20. Overall Results Average 95% Confidence Interval Length Median of 95% Confidence Interval Length

  21. Overall Results Percentage of Subjects for whom the bootstrapping confidence interval is wider than the likelihood ratio method confidence interval

  22. Likelihood Ratio Method vs Bootstrapping • Likelihood Ratio Method: • Is easy to compute • Very fast • BUT depends on assumptions that are usually violated by system dynamics models • Yields very tight confidence intervals • Bootstrapping: • Is NOT easy to compute • Takes longer time • DOES NOT depend on assumptions that are usually violated by system dynamics models • Yields larger confidence intervals. Usually more conservative.

More Related