Simulation Modeling and Analysis

1. Simulation Modeling and Analysis Output Analysis 1

2. Outline • Stochastic Nature of Output • Taxonomy of Simulation Outputs • Measures of Performance • Point Estimation • Interval Estimation • Output Analysis in Terminating Simulations • Output Analysis in Steady-state Simulations 2

3. Introduction • Output Analysis • Analysis of data produced by simulation • Goal • To predict system performance • To compare alternatives • Why is it needed? • To evaluate the precision of the simulation performance parameter as an estimator 3

4. Introduction -contd • Each simulation run is a sample point • Attempts to increase the sample size by increasing run length may fail because of autocorrelation • Initial conditions affect the output 4

5. Stochastic Nature of Output Data • Model Input Variables are Random Variables • The Model Transforms Input into Output • Output Data are Random Variables • Replications of a model run can be obtained by repeating the run using different random number streams 5

6. Example: M/G/1 Queue • Average arrival rate Poisson with  = 0.1 per minute • Service times Normal with  = 9.5 minutes and  = 1.75 minutes • Runs • One 5000 minute run • Five 1000 minute runs w/ 3 replications each 6

7. Taxonomy of Simulation Outputs • Terminating (Transient) Simulations • Runs until a terminating event takes place • Uses well specified initial conditions • Non-terminating (Steady-state) Simulations • Runs continually or over a very long time • Results must be independent of initial data • Termination? • What determines the type of simulation? 7

8. Examples: Non-terminating Systems • Many shifts of a widget manufacturing process. • Expansion in workload of a computer service bureau. 8

9. Measures of Performance: Point Estimation • Means • Proportions • Quantiles 9

10. Measures of Performance: Point Estimation (Discrete-time Data) • Point estimator of  (of ) based on the simulation discrete-time output (Y1, Y2,.., Yn) * = (1/n) i n Yi • Unbiased point estimator E(* ) =  • Bias b = E(* ) -  10

11. Measures of Performance: Point Estimation (Continuous-time data) • Point estimator of  (of ) based on the simulation continuous-time output (Y(t), 0 < t < Te) * = (1/ Te) 0 Te Y(t) dt • Unbiased point estimator E(* ) =  • Bias b = E(* ) -  11

12. Measures of Performance: Interval Estimation (Discrete-time Data) • Variance and variance estimator 2() = true variance of point estimator  2*() = estimator of variance of point estimator  • Bias (in variance estimation) B = E(2*() )/ 2() 12

13. Measures of Performance: Interval Estimation - contd • If B ~ 1 then t = ( - )/ 2*() has t/2,f distribution (d.o.f. = f). I.e. • A 100(1 - )% confidence interval for is  - t/2,f2*() <  <  + t/2,f2*() • Cases • Statistically independent observations • Statistically dependent observations (time series). 13

14. Measures of Performance: Interval Estimation - contd • Statistically independent observations • Sample variance S2 = i n (Yi -  )2/(n-1) • Unbiased estimator of 2() 2*() = S2 /n • Standard error of the point estimator  *() = S /n 14

15. Measures of Performance: Interval Estimation - contd • Statistically dependent observations • Variance of  2() = (1/n2) i n j n cov(Yi , Yj ) • Lag k autocovariance k = cov(Yi , Yi+k ) • Lag k autocorrelation k = k0 15

16. Measures of Performance: Interval Estimation - contd • Statistically dependent observations (contd) • Variance of  2() = (0 /n) [ 1 + 2k=1 n-1 (1- k/n) k] = (0 /n) c • Positively autocorrelated time series (k > 0) • Negatively autocorrelated time series (k < 0) • Bias (in variance estimation) B = E(S2/n )/ 2() = (n/c - 1)/(n-1) 16

17. Measures of Performance: Interval Estimation - contd • Statistically dependent observations (contd) • Cases • Independent data k = 0, c = 1, B = 1 • Positively correlated data k > 0, c > 1, B < 1, S2/n is biased low (underestimation) • Negatively correlated data k < 0, c < 1, B > 1, S2/n is biased high (overestimation) 17

18. Output Analysis for Terminating Simulations • Method of independent replications • n = Sample size • Number of replications r=1,2,…,R • Yji i-th observation in replication j • Yji, Yjk are autocorrelated • Yri, Ysk are statistically independent • Estimator of mean (r =1,2,…,R) r(1/nr) i nrYri 18

19. Output Analysis for Terminating Simulations - contd • Confidence Interval (R fixed; discrete data) • Overall point estimate * = (1/R) 1 Rr • Variance estimate * (*) = [1/(R-1)R] 1 R(r • Standard error of the point estimator  *() = * (*) 19

20. Output Analysis for Terminating Simulations - contd • Estimator and Interval (R fixed; continuous data) • Estimator of mean (r =1,2,…,R) r(1/Te) 0 Te Yr(t) dt Overall point estimate * = (1/R) 1 Rr • Variance estimate * (*) = [1/(R-1)R] 1 R(r 20

21. Output Analysis in Terminating Simulations - contd • Confidence Intervals with Specified Precision • Half-length confidence interval (h.l.) h.l. = t/2,f2*() = t/2,f S/ R <  • Required number of replications R* > ( z /2 So/  )2 21

22. Output Analysis for Steady State Simulations • Let (Y1, Y2,.., Yn) be an autocorrelated time series • Estimator of the long run measure of performance  (independent of I.C.s)  = lim n => (1/n) i n Yi • Sample size n (or Te) is design choice. 22

23. Output Analysis for Steady State Simulations -contd • Considerations affecting the choice of n • Estimator bias due to initial conditions • Desired precision of point estimator • Budget/computer constraints 23

24. Output Analysis for Steady State Simulations -contd • Initialization bias and Initialization methods • Intelligent initialization • Using actual field data • Using data from a simpler model • Use of phases in simulation • Initialization phase (0 < t < To; for i=1,2,…,d) • Data collection phase (To < t < Te; for i=d+1,d+2,…,n) • Rule of thumb (n-d) > 10 d 24

25. Output Analysis for Steady State Simulations -contd • Example M/G/1 queue • Batched data • Batched means • Averaging batch means within a replication (I.e. along the batches) • Averaging batch means within a batch (I.e. along the replications). 25

26. Steady State Simulations: Replication Method • Cases 1.- Yrj is an individual observation from within a replication 2.- Yrj is a batch mean of discrete data from within a replication 3.- Yrj is a batch mean of continuous data over a given interval 26

27. Steady State Simulations: Replication Method -contd • Sample average for replication r of all (nondeleted) observations Y*r(n,d) = Y*r = [1/(n-d)] j=d+1n Yrj • Replication averages are independent and identically distributed RV’s • Overall point estimator Y*(n,d) = Y* = [1/R] r=1R Yr(n,d) 27

28. Steady State Simulations: Replication Method -contd • Sample Variance S2 = [1/(R-1)] r=1R (Y*r - Y*) • Standard error = S/ R • 100(1-)% Confidence interval Y* - t /2,R-1 S/ R <  < Y* + t /2,R-1 S/ R 28

29. Steady State Simulations: Sample Size • Greater precision can be achieved by • Increasing the run length • Increasing the number of replications 29

30. Steady State Simulations: Batch Means for Interval Estimation • Single, long replication with batches • Batch means treated as if they were independent • Batch means (continuous) Y*j = (1/m) (j-1)mjm Y(t) dt • Batch means (discrete) Y*j = (1/m) i=(j-1)mjm Yi 30

31. Steady State Simulations: Batch Size Selection Guidelines • Number of batches < 30 • Diagnose correlation with lag 1 autocorrelation obtained from a large number of batch means from a smaller batch size • For total sample size to be selected sequentially allow batch size and number of batches grow with run length. 31