Output Analysis for Simulation

1. Output Analysis for Simulation Written by: Marvin K. Nakayama Presented by: Jennifer Burke MSIM 752

2. Outline • Performance measures • Output of a transient simulation • Techniques for steady-state simulations • Estimation of multiple performance measures • Other methods for analyzing simulation output

3. Example • Automatic Teller Machine (ATM)

4. Performance Measures • Measure how well the simulation runs • Different types of simulations require different statistical techniques to analyze the results • Terminating (or transient) • Steady-state (or long run)

5. Terminating Performance Measures • Terminating simulation • Simulation will finish at a given event • Initial conditions have a large impact • Ex: Queue starts with no customers present

6. ATM example (Terminating) • Open 9:00am – 5:00pm • X = # of customers using ATM in a day • E(X) • P(X  500) • C = queue is empty

7. Output of a Terminating Simulation • Goal: • calculate E(X) • Approach: • n 2 i.i.d duplications • X1,X2,…,Xn • find the average of those duplications

8. Output of a Terminating Simulation • calculate the sample variance of X1,X2,…,Xn and the sample standard deviation

9. Output of a Terminating Simulation • Central Limit Theorem • confidence interval for E(X)

10. Output of a Terminating Simulation • the confidence interval provides a form of error bound Hn is the half-width of the confidence interval

11. ATM example (Terminating) • Expected daily withdraw • within $500 • ε = 500 • S(n) = sample standard deviation

12. Steady-state Performance Measures • Steady-state simulation • Simulation that stabilizes over time • Initial condition C • Fi(y|C) • Fi(y|C) → F(y) as i → 

13. ATM example (Steady-state) • Open 24 hours a day • Yi = number of customers served on the ith day of operation • E(Y) • P(Y  400) • C = queue is empty

14. Output of a Steady-state Simulation • Case 1: • discrete-time process Y1,Y2,…,Yn • estimate v, as m →  • Case 2: • continuous-valued time index Y(s) • estimate v, as m → 

15. ATM (Continuous) • Y(s) = number of customers waiting in line at time s • Assume Y(s) has a steady-state • Calculate v

16. Difficulties of Steady-state analysis • Discrete-time process • if m is large, then is a good approximation of v • Confidence interval

17. Simplifications to Steady-state Analysis • Multiple replications • Initial-data deletion • Single-replicate algorithm

18. Method of Multiple Replications • Estimate • r i.i.d replications, length k = m/r • 10 r 30

19. Method of Multiple Replications • Average of jth row • Using find the sample mean

20. Method of Multiple Replications • Sample variance • Confidence interval

21. Problem with Multiple Replication Method • Simple estimation of variance • can be contaminated by initialization bias

22. Initial-Data Deletion • Partial solution • Delete first c observations • Replication mean sample mean sample variance confidence interval

23. Single-Replicate Algorithm • Single simulation of length m + c • Divide the m observations into n batches • 10 n 30 • Batch mean

24. Single-Replicate Algorithm • Sample mean • Sample variance • Confidence interval

25. Estimating Multiple Performance Measures • Terminating simulations • Confidence interval for each performance measure • Joint confidence interval

26. ATM example (Terminating) • Open 9:00am – 5:00pm • μ1 = expected # of customers served in a day • μ2= probability # served in a day is at least 1000 • μ3 = expected amount of $ withdrawn in a day

27. Conclusions • Basics of analyzing simulation output • Application potential is high • Not state of the art • Benefit • Lacked comparison